v0.4.8 Release (#720)

* Bump versions by dotnet-bump-version. * Bump versions by dotnet-bump-version. * Workflow updates (#658) # Added - Added: Added automatic character parsing for discord notifier. Now if the PR is over a certain character limit, it will trim and add an appropriate link to the full changelog. (Release for Stable, PR for Dev) # Removed - Removed: Removed Sentry map task from the workflow since Sentry is no longer used. * Bump versions by dotnet-bump-version. * Misc Updates (#665) * Do not allow non-admins to change their passwords when authentication is disabled * Clean up the login page so that input field text is black * cleanup some resizing when typing a password and having a lot of users * Changed the LastActive for a user to not just be login, but also when they open an already authenticated session. * Bump versions by dotnet-bump-version. * Logging Cleanup (#668) * Do not allow non-admins to change their passwords when authentication is disabled * Clean up the login page so that input field text is black * cleanup some resizing when typing a password and having a lot of users * Changed the LastActive for a user to not just be login, but also when they open an already authenticated session. * Removed some verbose debugging statements and moved some debug to information to be more prevelant to logs for default installs. * In Progress now sends progress information on the Series * Add ability to add cards to recently added when new series are added in backend * Implemented the ability to click the glasses icon to turn off incognito mode from within the reader so you can start tracking progress * Don't warn the user about authentication when they don't touch that control * Bump versions by dotnet-bump-version. * Changed the stats that are sent back to stat server from installed server. * Revert "Changed the stats that are sent back to stat server from installed server." This reverts commit 644cb6d1f6. * Bump versions by dotnet-bump-version. * Bump versions by dotnet-bump-version. * Bulk Add to Collection (#674) * Fixed the typeahead not having the same size input box as other inputs * Implemented the ability to add multiple series to a collection through bulk operations flow. Updated book parser to handle "@import url('...');" syntax as well as @import '...'; * Implemented the ability to create a new Collection tag via bulk operations flow. * Bump versions by dotnet-bump-version. * Bulk Operations for In Progress and Recently Added (#677) * Don't log a message about bad match if the file is a cover image * Enable bulk operations for In Progress and Recently Added * Fixed a bad logic case * Bump versions by dotnet-bump-version. * Regression Fix (#680) * Ensure we mount the backups directory for Docker users * Fixed a huge logic bug that deleted files in users libraries * Bump versions by dotnet-bump-version. * Change chunk size to be a fixed 50 to validate if it's causing issue with refresh. Added some try catches to see if exceptions are causing issues. (#681) * Bump versions by dotnet-bump-version. * Fixed a bug where searching on localized name would fail to show on the search. Fixed a bug where extra spaces would cause the search results not to show properly. (#682) * Bump versions by dotnet-bump-version. * When we have a special marker, ensure we fall back to folder parsing to try and group correctly to the actual series before just accepting what we parsed. (#684) Fixed a missed parsing case where comic special parsing wasn't being called on comic libraries. * Bump versions by dotnet-bump-version. * iOS Admin page dropdown fix (#686) # Fixed: - Fixed: Fixed an issue where the dropdown on the admin server page would not work on Safari or other iOS browsers. * When the DB fails to save, log out all the series the user should look into for constraint issues and push a message to the admins connected to webui. (#687) * Bump versions by dotnet-bump-version. * Bump versions by dotnet-bump-version. * Stat upload will now schedule itself between midnight and 6am in server time for upload. (#688) * Bump versions by dotnet-bump-version. * EPUB CSS Parsing Issues (#690) * WIP. Rewrote some of the Regex to better support css escaping. We now escape background-image, border-image, and list-style-image within css files. * Added position relative to help with positioning on books that are just absolute positioned elements. * When there is absolute positioning, like in some epub based comics, supress the bottom action bar since it wont render in the correct location. * Fixed tests * Commented out tests * Bump versions by dotnet-bump-version. * More EPUB Scoping Fixes (#691) * Added better handling around when importing css files that are empty. Moved comment removal on css files to before some css whitespace cleanup to get better matches. * Some enhancements on the checks to see if we need the bottom action bar on reader. Now we don't query DOM and have something that works more reliably. * Bump versions by dotnet-bump-version. * Fixed an issue where docker users were not properly backing up the database. Removed an empty File for when covers/ had nothing in it. (#692) * Bump versions by dotnet-bump-version. * Fallback to Folder Parsing Issue (#694) * Fixed a bug in the scanner where we fall back to parsing from folders for poorly named files. The code was exiting early if a chapter or volume could be parsed out. * Fixed a unit test by tweaking a regex for fallback * Bump versions by dotnet-bump-version. * KavitaStats Cleanup (#695) * Refactored Stats code to be much cleaner and user better naming. * Cleaned up the actual http code to use Flurl and to return if the upload was successful or not so we can delete the file where appropriate. * More refactoring for the stats code to clean it up and keep it consistent with our standards. * Removed a confusing log statement * Added support for old api key header from original stat server * Use the correct endpoint, not the new one. * Code smell * Bump versions by dotnet-bump-version. * Bulk Deletion (#697) * Implemented bulk deletion of series * Don't show unauthorized exception on UI, just redirect to the login page. * Bump versions by dotnet-bump-version. * Cover Image Picking + Forwarding Headers with EPUBs (#700) * Ensure Kavita knows about forwarding headers (fixes issue with epub urls not going through https with reverse proxy). Fixed a case where cover image selection preferred nested folders vs files in root directory. * Fixed broken unit test * Added bug that I fixed to the unit tests * Cover Image Picking + Forwarding Headers with EPUBs (#702) * Updating GA Bump version temporarily for fix (#703) * Bump versions by dotnet-bump-version. * Cover Image Picking + Forwarding Headers with EPUBs (GA Fix) (#704) * Bump versions by dotnet-bump-version. * Vacation Fixes (#709) * Ignore system and hidden folders when performing directory scan. * Fixed the comic parser tests not using Comic mode for parsing. * Accept all forwarded headers and use them. * Ignore some changes from another branch * Bump versions by dotnet-bump-version. * Breaking Changes: Docker Parity (#698) * Refactored all the config files for Kavita to be loaded from config/. This will allow docker to just mount one folder and for Update functionality to be trivial. * Cleaned up documentation around new update method. * Updated docker files to support config directory * Removed entrypoint, no longer needed * Update appsettings to point to config directory for logs * Updated message for docker users that are upgrading * Ensure that docker users that have not updated their mount points from upgrade cannot start the server * Code smells * More cleanup * Added entrypoint to fix bind mount issues * Updated README with new folder structure * Fixed build system for new setup * Updated string path if user is docker * Updated the migration flow for docker to work properly and Fixed LogFile configuration updating. * Migrating docker images is now working 100% * Fixed config from bad code * Code cleanup Co-authored-by: Chris Plaatjes <kizaing@gmail.com> * Bump versions by dotnet-bump-version. * Feature/docker parity (#714) * Refactored all the config files for Kavita to be loaded from config/. This will allow docker to just mount one folder and for Update functionality to be trivial. * Cleaned up documentation around new update method. * Updated docker files to support config directory * Removed entrypoint, no longer needed * Update appsettings to point to config directory for logs * Updated message for docker users that are upgrading * Ensure that docker users that have not updated their mount points from upgrade cannot start the server * Code smells * More cleanup * Added entrypoint to fix bind mount issues * Updated README with new folder structure * Fixed build system for new setup * Updated string path if user is docker * Updated the migration flow for docker to work properly and Fixed LogFile configuration updating. * Migrating docker images is now working 100% * Fixed config from bad code * Code cleanup * Fixed monorepo-build.sh Co-authored-by: Chris Plaatjes <kizaing@gmail.com> * Breaking Changes: Docker Parity (#715) * Fixed a bug in the copy directory to directory in the migration * Somehow GetFiles lost static modifier. * Bump versions by dotnet-bump-version. * Build issue (#716) * Fixed a bug in the copy directory to directory in the migration * Somehow GetFiles lost static modifier. * Please work * Bump versions by dotnet-bump-version. * Bump versions by dotnet-bump-version. * Shakeout Changes (#717) * Make the appsettings public on Configuration and change how we detect when to migrate for non-docker users. * Fixed up non-docker copy command and removed duplicate check on source directory for a copy. * Don't delete files unless we know we are successful * Bump versions by dotnet-bump-version. * Fixed a migration issue on docker happening too many times or throwing exception when source wasn't there. (#719) * Bump versions by dotnet-bump-version. * Version bump for release (#718) * Bump versions by dotnet-bump-version. Co-authored-by: Robbie Davis <robbie@therobbiedavis.com> Co-authored-by: YEGCSharpDev <89283498+YEGCSharpDev@users.noreply.github.com> Co-authored-by: Chris Plaatjes <kizaing@gmail.com>
2021-11-04 07:29:02 -05:00 · 2021-11-04 07:29:02 -05:00 · 33db123e81
commit 33db123e81
parent cb9fa0dda8
115 changed files with 1818 additions and 910 deletions
--- a/API/Data/MigrateConfigFiles.cs
+++ b/API/Data/MigrateConfigFiles.cs
@ -0,0 +1,166 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using API.Services;
+using Kavita.Common;
+
+namespace API.Data
+{
+    public static class MigrateConfigFiles
+    {
+        private static readonly List<string> LooseLeafFiles = new List<string>()
+        {
+            "appsettings.json",
+            "appsettings.Development.json",
+            "kavita.db",
+        };
+
+        private static readonly List<string> AppFolders = new List<string>()
+        {
+            "covers",
+            "stats",
+            "logs",
+            "backups",
+            "cache",
+            "temp"
+        };
+
+        private static readonly string ConfigDirectory = Path.Join(Directory.GetCurrentDirectory(), "config");
+
+
+        /// <summary>
+        /// In v0.4.8 we moved all config files to config/ to match with how docker was setup. This will move all config files from current directory
+        /// to config/
+        /// </summary>
+        public static void Migrate(bool isDocker)
+        {
+            Console.WriteLine("Checking if migration to config/ is needed");
+
+            if (isDocker)
+            {
+                if (Configuration.LogPath.Contains("config"))
+                {
+                    Console.WriteLine("Migration to config/ not needed");
+                    return;
+                }
+
+                Console.WriteLine(
+                    "Migrating files from pre-v0.4.8. All Kavita config files are now located in config/");
+
+                CopyAppFolders();
+                DeleteAppFolders();
+
+                UpdateConfiguration();
+
+                Console.WriteLine("Migration complete. All config files are now in config/ directory");
+                return;
+            }
+
+            if (new FileInfo(Configuration.AppSettingsFilename).Exists)
+            {
+                Console.WriteLine("Migration to config/ not needed");
+                return;
+            }
+
+            Console.WriteLine(
+                "Migrating files from pre-v0.4.8. All Kavita config files are now located in config/");
+
+            Console.WriteLine($"Creating {ConfigDirectory}");
+            DirectoryService.ExistOrCreate(ConfigDirectory);
+
+            try
+            {
+                CopyLooseLeafFiles();
+
+                CopyAppFolders();
+
+                // Then we need to update the config file to point to the new DB file
+                UpdateConfiguration();
+            }
+            catch (Exception)
+            {
+                Console.WriteLine("There was an exception during migration. Please move everything manually.");
+                return;
+            }
+
+            // Finally delete everything in the source directory
+            Console.WriteLine("Removing old files");
+            DeleteLooseFiles();
+            DeleteAppFolders();
+            Console.WriteLine("Removing old files...DONE");
+
+            Console.WriteLine("Migration complete. All config files are now in config/ directory");
+        }
+
+        private static void DeleteAppFolders()
+        {
+            foreach (var folderToDelete in AppFolders)
+            {
+                if (!new DirectoryInfo(Path.Join(Directory.GetCurrentDirectory(), folderToDelete)).Exists) continue;
+
+                DirectoryService.ClearAndDeleteDirectory(Path.Join(Directory.GetCurrentDirectory(), folderToDelete));
+            }
+        }
+
+        private static void DeleteLooseFiles()
+        {
+            var configFiles = LooseLeafFiles.Select(file => new FileInfo(Path.Join(Directory.GetCurrentDirectory(), file)))
+                .Where(f => f.Exists);
+            DirectoryService.DeleteFiles(configFiles.Select(f => f.FullName));
+        }
+
+        private static void CopyAppFolders()
+        {
+            Console.WriteLine("Moving folders to config");
+
+                foreach (var folderToMove in AppFolders)
+                {
+                    if (new DirectoryInfo(Path.Join(ConfigDirectory, folderToMove)).Exists) continue;
+
+                    try
+                    {
+                        DirectoryService.CopyDirectoryToDirectory(
+                            Path.Join(Directory.GetCurrentDirectory(), folderToMove),
+                            Path.Join(ConfigDirectory, folderToMove));
+                    }
+                    catch (Exception)
+                    {
+                        /* Swallow Exception */
+                    }
+                }
+
+
+            Console.WriteLine("Moving folders to config...DONE");
+        }
+
+        private static void CopyLooseLeafFiles()
+        {
+            var configFiles = LooseLeafFiles.Select(file => new FileInfo(Path.Join(Directory.GetCurrentDirectory(), file)))
+                .Where(f => f.Exists);
+            // First step is to move all the files
+            Console.WriteLine("Moving files to config/");
+            foreach (var fileInfo in configFiles)
+            {
+                try
+                {
+                    fileInfo.CopyTo(Path.Join(ConfigDirectory, fileInfo.Name));
+                }
+                catch (Exception)
+                {
+                    /* Swallow exception when already exists */
+                }
+            }
+
+            Console.WriteLine("Moving files to config...DONE");
+        }
+
+        private static void UpdateConfiguration()
+        {
+            Console.WriteLine("Updating appsettings.json to new paths");
+            Configuration.DatabasePath = "config//kavita.db";
+            Configuration.LogPath = "config//logs/kavita.log";
+            Console.WriteLine("Updating appsettings.json to new paths...DONE");
+        }
+    }
+}
--- a/API/Data/Repositories/CollectionTagRepository.cs
+++ b/API/Data/Repositories/CollectionTagRepository.cs
@ -3,6 +3,7 @@ using System.IO;
 using System.Linq;
 using System.Threading.Tasks;
 using API.DTOs;
+using API.DTOs.CollectionTags;
 using API.Entities;
 using API.Interfaces.Repositories;
 using AutoMapper;
@ -22,6 +23,11 @@ namespace API.Data.Repositories
            _mapper = mapper;
        }

+        public void Add(CollectionTag tag)
+        {
+            _context.CollectionTag.Add(tag);
+        }
+
        public void Remove(CollectionTag tag)
        {
            _context.CollectionTag.Remove(tag);
--- a/API/Data/Repositories/SeriesMetadataRepository.cs
+++ b/API/Data/Repositories/SeriesMetadataRepository.cs
@ -0,0 +1,20 @@
+using API.Entities;
+using API.Interfaces.Repositories;
+
+namespace API.Data.Repositories
+{
+    public class SeriesMetadataRepository : ISeriesMetadataRepository
+    {
+        private readonly DataContext _context;
+
+        public SeriesMetadataRepository(DataContext context)
+        {
+            _context = context;
+        }
+
+        public void Update(SeriesMetadata seriesMetadata)
+        {
+            _context.SeriesMetadata.Update(seriesMetadata);
+        }
+    }
+}
--- a/API/Data/Repositories/SeriesRepository.cs
+++ b/API/Data/Repositories/SeriesRepository.cs
@ -4,6 +4,7 @@ using System.Linq;
 using System.Threading.Tasks;
 using API.Data.Scanner;
 using API.DTOs;
+using API.DTOs.CollectionTags;
 using API.DTOs.Filtering;
 using API.Entities;
 using API.Extensions;
@ -41,6 +42,11 @@ namespace API.Data.Repositories
            _context.Series.Remove(series);
        }

+        public void Remove(IEnumerable<Series> series)
+        {
+            _context.Series.RemoveRange(series);
+        }
+
        public async Task<bool> DoesSeriesNameExistInLibrary(string name)
        {
            var libraries = _context.Series
@ -171,6 +177,21 @@ namespace API.Data.Repositories
                .SingleOrDefaultAsync();
        }

+        /// <summary>
+        /// Returns Volumes, Metadata, and Collection Tags
+        /// </summary>
+        /// <param name="seriesIds"></param>
+        /// <returns></returns>
+        public async Task<IList<Series>> GetSeriesByIdsAsync(IList<int> seriesIds)
+        {
+            return await _context.Series
+                .Include(s => s.Volumes)
+                .Include(s => s.Metadata)
+                .ThenInclude(m => m.CollectionTags)
+                .Where(s => seriesIds.Contains(s.Id))
+                .ToListAsync();
+        }
+
        public async Task<int[]> GetChapterIdsForSeriesAsync(int[] seriesIds)
        {
            var volumes = await _context.Volume
@ -454,15 +475,15 @@ namespace API.Data.Repositories
            // TODO: Think about making this bigger depending on number of files a user has in said library
            // and number of cores and amount of memory. We can then make an optimal choice
            var totalSeries = await GetSeriesCount(libraryId);
-            var procCount = Math.Max(Environment.ProcessorCount - 1, 1);
-
-            if (totalSeries < procCount * 2 || totalSeries < 50)
-            {
-                return new Tuple<int, int>(totalSeries, totalSeries);
-            }
-
-
-            return new Tuple<int, int>(totalSeries, Math.Max(totalSeries / procCount, 50));
+            // var procCount = Math.Max(Environment.ProcessorCount - 1, 1);
+            //
+            // if (totalSeries < procCount * 2 || totalSeries < 50)
+            // {
+            //     return new Tuple<int, int>(totalSeries, totalSeries);
+            // }
+            //
+            // return new Tuple<int, int>(totalSeries, Math.Max(totalSeries / procCount, 50));
+            return new Tuple<int, int>(totalSeries, 50);
        }

        public async Task<Chunk> GetChunkInfo(int libraryId = 0)
@ -485,5 +506,13 @@ namespace API.Data.Repositories
                TotalChunks = totalChunks
            };
        }
+
+        public async Task<IList<SeriesMetadata>> GetSeriesMetadataForIdsAsync(IEnumerable<int> seriesIds)
+        {
+            return await _context.SeriesMetadata
+                .Where(sm => seriesIds.Contains(sm.SeriesId))
+                .Include(sm => sm.CollectionTags)
+                .ToListAsync();
+        }
    }
 }
--- a/API/Data/Repositories/SettingsRepository.cs
+++ b/API/Data/Repositories/SettingsRepository.cs
@ -35,15 +35,6 @@ namespace API.Data.Repositories
            return _mapper.Map<ServerSettingDto>(settings);
        }

-        public ServerSettingDto GetSettingsDto()
-        {
-            var settings = _context.ServerSetting
-                .Select(x => x)
-                .AsNoTracking()
-                .ToList();
-            return _mapper.Map<ServerSettingDto>(settings);
-        }
-
        public Task<ServerSetting> GetSettingAsync(ServerSettingKey key)
        {
            return _context.ServerSetting.SingleOrDefaultAsync(x => x.Key == key);
--- a/API/Data/Seed.cs
+++ b/API/Data/Seed.cs
@ -41,11 +41,11 @@ namespace API.Data

            IList<ServerSetting> defaultSettings = new List<ServerSetting>()
            {
-                new() {Key = ServerSettingKey.CacheDirectory, Value = CacheService.CacheDirectory},
+                new() {Key = ServerSettingKey.CacheDirectory, Value = DirectoryService.CacheDirectory},
                new () {Key = ServerSettingKey.TaskScan, Value = "daily"},
                new () {Key = ServerSettingKey.LoggingLevel, Value = "Information"}, // Not used from DB, but DB is sync with appSettings.json
                new () {Key = ServerSettingKey.TaskBackup, Value = "weekly"},
-                new () {Key = ServerSettingKey.BackupDirectory, Value = Path.GetFullPath(Path.Join(Directory.GetCurrentDirectory(), "backups/"))},
+                new () {Key = ServerSettingKey.BackupDirectory, Value = Path.GetFullPath(DirectoryService.BackupDirectory)},
                new () {Key = ServerSettingKey.Port, Value = "5000"}, // Not used from DB, but DB is sync with appSettings.json
                new () {Key = ServerSettingKey.AllowStatCollection, Value = "true"},
                new () {Key = ServerSettingKey.EnableOpds, Value = "false"},
@ -69,6 +69,8 @@ namespace API.Data
                Configuration.Port + string.Empty;
            context.ServerSetting.First(s => s.Key == ServerSettingKey.LoggingLevel).Value =
                Configuration.LogLevel + string.Empty;
+            context.ServerSetting.First(s => s.Key == ServerSettingKey.CacheDirectory).Value =
+                DirectoryService.CacheDirectory + string.Empty;

            await context.SaveChangesAsync();

--- a/API/Data/UnitOfWork.cs
+++ b/API/Data/UnitOfWork.cs
@ -34,6 +34,7 @@ namespace API.Data
        public IFileRepository FileRepository => new FileRepository(_context);
        public IChapterRepository ChapterRepository => new ChapterRepository(_context, _mapper);
        public IReadingListRepository ReadingListRepository => new ReadingListRepository(_context, _mapper);
+        public ISeriesMetadataRepository SeriesMetadataRepository => new SeriesMetadataRepository(_context);

        /// <summary>
        /// Commits changes to the DB. Completes the open transaction.