AZEN-SGG/Kavita
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Bugs, Enhancements, and Performance (#580)

Browse source 
* Added parser case for "The Duke of Death and His Black Maid - Ch. 177 - The Ball (3).cbz"

* Removed a file that is created and modified every test run.

* Fixed a bad parser case for "Batman Beyond 02 (of 6) (1999)" which was consuming too many characters

* Removed a lot of "Volume" parsing for Comics that don't make sense. This is prep work for the upcoming Comic Rework release.

* Reworked a lot of parsing cases for comics based on naming conventions observed from releases found online.

* Added a way for external scripts to use a user api key to authenticate

* Fixed an issue if the manga only had one page, the bottom menu would be missing page and chapter controls.

* Fixed a bug where on small phones, nav bar could overflow due to scroll to top

* Tweaked a lot of regex for manga parsing to handle some cases where poorly named files, like "Vol. 03 Ch. 21" would end up parsing as Series "Vol. 03".

* Even more handling of parser cases. Manga parser should be as it was but more robust to handle bad naming.

* Fixed: Don't force metadata refresh on Scan Series, only on refresh metadata

* Implemented the ability to automatically refresh after a series scan based on when server finishes. Remove a duplicate API call from series detail.

* Removed another API call for series metadata that isn't needed.

* Refactored Message creation to a factory, hardcoded strings are centralized, and RefreshSeriesMetadata sends an event and is refactored to be async.

* Fixed a bug when really poorly named files are within a folder that contains the series name, fallback couldn't occur due to it being taken as root folder. Now we detect said condition and will go one level higher, resulting in potentially more I/O, but the series will not be deleted.

* Added the Read in Incognito context item for Chapter cards

* Skip an additional check for series summary for series that aren't EPUB or Archive formats.

* Fixed an issue where cover image generation could occur due to a bad check on LastWriteTime on the underlying file.

* Added some extra comic parser tests

* Added a ScanLibrary event (not hooked up in UI)

* Performance improvement on metadata service. Now when we scan for cover image changes, we emit when a change occurs and only then do we update parent entities (array copy).

* Removed an hr from series detail and ensure we update the cover image for series when scan series finishes.

* Updated the infinite scroller to use a Flags pattern for the debug mode. Updated a few logical conditions for mobile.

* Removed the concurrency check on row progress as if too many calls hit the DB, it will throw, but it doesn't matter.

Fixed a bad logic code which could cause scrolling after hitting the bottom of the chapter.

* Ensure prefetching uses totalPages + 1 since we pass in totalPages as - 1 from manga reader

* Fixed issue where last page of webtoon wouldn't be prefetched due to a < instead of <= on prefetching code

* Implemented ability to send images from archives to the UI without incurring any extra memory pressure.

* Dropdown menus now have a darker background

* Webtoon reader now works on mobile.

* Fixed how keyboard presses for up/down/left/right work with MANGA_UD reading mode. See issue #579

* Fixed cont reader for webtoons on mobile

* Fixed a small issue where top spacer would too quickly switch to prev chapter

* Updated user preferences to use same slider style. Removed some css that is not used.

* Added comic parser case for "Saga 001 (2012) (Digital) (Empire-Zone)"

* Added accessibility toggle to reading list order and aligned sliders to all use the same style.

* Removed a todo for checking on new image serving code. It works great.

* Fixed a missing await

* Auth guard will now check if an existing toast is present giving same message before poping the toast.

* Fixed alignment on phones for reading lists

* Moved sorters so they aren't resused between multiple threads. Slightly higher memory footprint.

* Fixed a broken unit test

* Code smells

* More unit test fixing
This commit is contained in:
Joseph Milazzo 2021-09-15 11:06:29 -07:00 • committed by GitHub
parent b62d581491
commit cf4fd2cb9c
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
52 changed files with 685 additions and 336 deletions
Download patch file Download diff file Expand all files Collapse all files

86
API/Services/Tasks/ScannerService.cs
View file

@ -14,7 +14,9 @@ using API.Interfaces;
using API.Interfaces.Services;
using API.Parser;
using API.Services.Tasks.Scanner;
using API.SignalR;
using Hangfire;
using Microsoft.AspNetCore.SignalR;
using Microsoft.Extensions.Logging;
namespace API.Services.Tasks
@ -27,10 +29,11 @@ namespace API.Services.Tasks
private readonly IMetadataService _metadataService;
private readonly IBookService _bookService;
private readonly ICacheService _cacheService;
private readonly IHubContext<MessageHub> _messageHub;
private readonly NaturalSortComparer _naturalSort = new ();
public ScannerService(IUnitOfWork unitOfWork, ILogger<ScannerService> logger, IArchiveService archiveService,
IMetadataService metadataService, IBookService bookService, ICacheService cacheService)
IMetadataService metadataService, IBookService bookService, ICacheService cacheService, IHubContext<MessageHub> messageHub)
{
_unitOfWork = unitOfWork;
_logger = logger;
@ -38,6 +41,7 @@ namespace API.Services.Tasks
_metadataService = metadataService;
_bookService = bookService;
_cacheService = cacheService;
_messageHub = messageHub;
}
[DisableConcurrentExecution(timeoutInSeconds: 360)]
@ -47,7 +51,7 @@ namespace API.Services.Tasks
var files = await _unitOfWork.SeriesRepository.GetFilesForSeries(seriesId);
var series = await _unitOfWork.SeriesRepository.GetSeriesByIdAsync(seriesId);
var library = await _unitOfWork.LibraryRepository.GetFullLibraryForIdAsync(libraryId, seriesId);
var dirs = FindHighestDirectoriesFromFiles(library, files);
var dirs = DirectoryService.FindHighestDirectoriesFromFiles(library.Folders.Select(f => f.Path), files.Select(f => f.FilePath).ToList());
var chapterIds = await _unitOfWork.SeriesRepository.GetChapterIdsForSeriesAsync(new []{ seriesId });
_logger.LogInformation("Beginning file scan on {SeriesName}", series.Name);
@ -63,6 +67,37 @@ namespace API.Services.Tasks
parsedSeries.Remove(key);
}
if (parsedSeries.Count == 0)
{
// We need to do an additional check for an edge case: If the scan ran and the files do not match the existing Series name, then it is very likely,
// the files have crap naming and if we don't correct, the series will get deleted due to the parser not being able to fallback onto folder parsing as the root
// is the series folder.
var existingFolder = dirs.Keys.FirstOrDefault(key => key.Contains(series.OriginalName));
if (dirs.Keys.Count == 1 && !string.IsNullOrEmpty(existingFolder))
{
dirs = new Dictionary<string, string>();
var path = Path.GetPathRoot(existingFolder);
if (!string.IsNullOrEmpty(path))
{
dirs[path] = string.Empty;
}
}
_logger.LogDebug("{SeriesName} has bad naming convention, forcing rescan at a higher directory.", series.OriginalName);
scanner = new ParseScannedFiles(_bookService, _logger);
parsedSeries = scanner.ScanLibrariesForSeries(library.Type, dirs.Keys, out var totalFiles2, out var scanElapsedTime2);
totalFiles += totalFiles2;
scanElapsedTime += scanElapsedTime2;
// If a root level folder scan occurs, then multiple series gets passed in and thus we get a unique constraint issue
// Hence we clear out anything but what we selected for
firstSeries = library.Series.FirstOrDefault();
keys = parsedSeries.Keys;
foreach (var key in keys.Where(key => !firstSeries.NameInParserInfo(parsedSeries[key].FirstOrDefault()) || firstSeries?.Format != key.Format))
{
parsedSeries.Remove(key);
}
}
var sw = new Stopwatch();
UpdateLibrary(library, parsedSeries);
@ -74,8 +109,10 @@ namespace API.Services.Tasks
totalFiles, parsedSeries.Keys.Count, sw.ElapsedMilliseconds + scanElapsedTime, series.Name);
CleanupDbEntities();
BackgroundJob.Enqueue(() => _metadataService.RefreshMetadataForSeries(libraryId, seriesId));
BackgroundJob.Enqueue(() => _metadataService.RefreshMetadataForSeries(libraryId, seriesId, forceUpdate));
BackgroundJob.Enqueue(() => _cacheService.CleanupChapters(chapterIds));
// Tell UI that this series is done
await _messageHub.Clients.All.SendAsync(SignalREvents.ScanSeries, MessageFactory.ScanSeriesEvent(seriesId), cancellationToken: token);
}
else
{
@ -83,54 +120,18 @@ namespace API.Services.Tasks
"There was a critical error that resulted in a failed scan. Please check logs and rescan");
await _unitOfWork.RollbackAsync();
}
}
/// <summary>
/// Finds the highest directories from a set of MangaFiles
/// </summary>
/// <param name="library"></param>
/// <param name="files"></param>
/// <returns></returns>
private static Dictionary<string, string> FindHighestDirectoriesFromFiles(Library library, IList<MangaFile> files)
{
var stopLookingForDirectories = false;
var dirs = new Dictionary<string, string>();
foreach (var folder in library.Folders)
{
if (stopLookingForDirectories) break;
foreach (var file in files)
{
if (!file.FilePath.Contains(folder.Path)) continue;
var parts = DirectoryService.GetFoldersTillRoot(folder.Path, file.FilePath).ToList();
if (parts.Count == 0)
{
// Break from all loops, we done, just scan folder.Path (library root)
dirs.Add(folder.Path, string.Empty);
stopLookingForDirectories = true;
break;
}
var fullPath = Path.Join(folder.Path, parts.Last());
if (!dirs.ContainsKey(fullPath))
{
dirs.Add(fullPath, string.Empty);
}
}
}
return dirs;
}
[DisableConcurrentExecution(timeoutInSeconds: 360)]
[AutomaticRetry(Attempts = 0, OnAttemptsExceeded = AttemptsExceededAction.Delete)]
public void ScanLibraries()
public async Task ScanLibraries()
{
var libraries = Task.Run(() => _unitOfWork.LibraryRepository.GetLibrariesAsync()).Result.ToList();
foreach (var lib in libraries)
{
ScanLibrary(lib.Id, false);
await ScanLibrary(lib.Id, false);
}
}
@ -145,7 +146,7 @@ namespace API.Services.Tasks
/// <param name="forceUpdate"></param>
[DisableConcurrentExecution(360)]
[AutomaticRetry(Attempts = 0, OnAttemptsExceeded = AttemptsExceededAction.Delete)]
public void ScanLibrary(int libraryId, bool forceUpdate)
public async Task ScanLibrary(int libraryId, bool forceUpdate)
{
Library library;
try
@ -188,6 +189,7 @@ namespace API.Services.Tasks
CleanupAbandonedChapters();
BackgroundJob.Enqueue(() => _metadataService.RefreshMetadata(libraryId, forceUpdate));
await _messageHub.Clients.All.SendAsync(SignalREvents.ScanLibrary, MessageFactory.ScanLibraryEvent(libraryId, "complete"));
}
/// <summary>

6
API/Services/Tasks/VersionUpdaterService.cs
View file

@ -140,11 +140,7 @@ namespace API.Services.Tasks
connections.AddRange(await _tracker.GetConnectionsForUser(admin));
}
await _messageHub.Clients.Users(admins).SendAsync("UpdateAvailable", new SignalRMessage
{
Name = "UpdateAvailable",
Body = update
});
await _messageHub.Clients.Users(admins).SendAsync(SignalREvents.UpdateVersion, MessageFactory.UpdateVersionEvent(update));
}