AZEN-SGG/Kavita
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Local Metadata Integration Part 1 (#817)

Browse source 
* Started with some basic plumbing with comic info parsing updating Series/Volume.

* We can now get chapter title from comicInfo.xml

* Hooked in the ability to store people into the chapter metadata.

* Removed no longer used imports, fixed up some foreign key constraints on deleting series with person linked.

* Refactored Summary out of the UI for Series into SeriesMetadata. Updated application to .net 6. There is a bug in metadata code for updating.

* Removed the parallel.ForEach with a normal foreach which lets us use async. For I/O heavy code, shouldn't change much.

* Refactored scan code to only check extensions with comic info, fixed a bug on scan events not using correct method name, removed summary field (still buggy)

* Fixed a bug where on cancelling a metadata request in modal, underlying button would get stuck in a disabled state.

* Changed how metadata selects the first volume to read summary info from. It will now select the first non-special volume rather than Volume 1.

* More debugging and found more bugs to fix

* Redid all the migrations as one single one. Fixed a bug with GetChapterInfo returning null when ChapterMetadata didn't exist for that Chapter.

Fixed an issue with mapper failing on GetChapterMetadata. Started work on adding people and a design for people.

* Fixed a bug where checking if file modified now takes into account if file has been processed at least once. Introduced a bug in saving people to series.

* Just made code compilable again

* Fixed up code. Now people for series and chapters add correctly without any db issues.

* Things are working, but I'm not happy with how the management of Person is. I need to take into account that 1 person needs to map to an image and role is arbitrary.

* Started adding UI code to showcase chapter metadata

* Updated workflow to be .NET 6

* WIP of updating card detail to show the information more clearly and without so many if statements

* Removed ChatperMetadata and store on the Chapter itself. Much easier to use and less joins.

* Implemented Genre on SeriesMetadata level

* Genres and People are now removed from Series level if they are no longer on comicInfo

* PeopleHelper is done with unit tests. Everything is working.

* Unit tests in place for Genre Helper

* Starting on CacheHelper

* Finished tests for ShouldUpdateCoverImage. Fixed and added tests in ArchiveService/ScannerService.

* CacheHelper is fully tested

* Some DI cleanup

* Scanner Service now calls GetComicInfo for books. Added ability to update Series Sort name from metadata files (mainly epub as comicinfo doesn't have a field)

* Forgot to move a line of code

* SortName now populates from metadata (epub only, ComicInfo has no tags)

* Cards now show the chapter title name if it's set on hover, else will default back to title.

* Fixed a major issue with how MangaFiles were being updated with LastModified, which messed up our logic for avoiding refreshes.

* Woohoo, more tests and some refactors to be able to test more services wtih mock filesystem. Fixed an issue where SortName was getting set as first chapter, but the Series was in a group.

* Refactored the MangaFile creation code into the DbFactory where we also setup the first LastModified update.

* Has file changed bug is now finally fixed

* Remove dead genres, refactor genre to use title instead of name.

* Refactored out a directory from ShouldUpdateCoverImage() to keep the code clean

* Unit tests for ComicInfo on BookService.

* Refactored series detail into it's own component

* Series-detail now received refresh metadata events to refresh what's on screen

* Removed references to Artist on PersonRole as it has no metadata mapping

* Security audit

* Fixed a benchmark

* Updated JWT Token generator to use new methods in .NET 6

* Updated all the docker and build commands to use net6.0

* Commented out sonar scan since it's not setup for net6.0 yet.
This commit is contained in:
Joseph Milazzo 2021-12-02 11:02:34 -06:00 • committed by GitHub
parent 10a6a3a544
commit e7619e6b0a
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
140 changed files with 9315 additions and 1545 deletions
Download patch file Download diff file Expand all files Collapse all files

4
API/Data/DataContext.cs
View file

@ -4,6 +4,7 @@ using System.Threading;
using System.Threading.Tasks;
using API.Entities;
using API.Entities.Interfaces;
using API.Entities.Metadata;
using Microsoft.AspNetCore.Identity;
using Microsoft.AspNetCore.Identity.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore;
@ -23,7 +24,6 @@ namespace API.Data
public DbSet<Library> Library { get; set; }
public DbSet<Series> Series { get; set; }
public DbSet<Chapter> Chapter { get; set; }
public DbSet<Volume> Volume { get; set; }
public DbSet<AppUser> AppUser { get; set; }
@ -37,6 +37,8 @@ namespace API.Data
public DbSet<AppUserBookmark> AppUserBookmark { get; set; }
public DbSet<ReadingList> ReadingList { get; set; }
public DbSet<ReadingListItem> ReadingListItem { get; set; }
public DbSet<Person> Person { get; set; }
public DbSet<Genre> Genre { get; set; }
protected override void OnModelCreating(ModelBuilder builder)

45
API/Data/DbFactory.cs
View file

@ -1,7 +1,11 @@
using System;
using System.Collections.Generic;
using System.IO;
using API.Data.Metadata;
using API.Entities;
using API.Entities.Enums;
using API.Entities.Metadata;
using API.Extensions;
using API.Parser;
using API.Services.Tasks;
@ -21,12 +25,16 @@ namespace API.Data
LocalizedName = name,
NormalizedName = Parser.Parser.Normalize(name),
SortName = name,
Summary = string.Empty,
Volumes = new List<Volume>(),
Metadata = SeriesMetadata(Array.Empty<CollectionTag>())
};
}
public static SeriesMetadata SeriesMetadata(ComicInfo info)
{
return SeriesMetadata(Array.Empty<CollectionTag>());
}
public static Volume Volume(string volumeNumber)
{
return new Volume()
@ -57,7 +65,8 @@ namespace API.Data
{
return new SeriesMetadata()
{
CollectionTags = collectionTags
CollectionTags = collectionTags,
Summary = string.Empty
};
}
@ -72,5 +81,37 @@ namespace API.Data
Promoted = promoted
};
}
public static Genre Genre(string name, bool external)
{
return new Genre()
{
Title = name.Trim().SentenceCase(),
NormalizedTitle = Parser.Parser.Normalize(name),
ExternalTag = external
};
}
public static Person Person(string name, PersonRole role)
{
return new Person()
{
Name = name.Trim(),
NormalizedName = Parser.Parser.Normalize(name),
Role = role
};
}
public static MangaFile MangaFile(string filePath, MangaFormat format, int pages)
{
return new MangaFile()
{
FilePath = filePath,
Format = format,
Pages = pages,
LastModified = DateTime.Now //File.GetLastWriteTime(filePath)
};
}
}
}

10
API/Data/Metadata/ComicInfo.cs
View file

@ -34,11 +34,19 @@
public string AlternativeSeries { get; set; }
public string AlternativeNumber { get; set; }
/// <summary>
/// This is Epub only: calibre:title_sort
/// Represents the sort order for the title
/// </summary>
public string TitleSort { get; set; }
/// <summary>
/// This is the Author. For Books, we map creator tag in OPF to this field. Comma separated if multiple.
/// </summary>
public string Writer { get; set; } // TODO: Validate if we should make this a list of writers
public string Writer { get; set; }
public string Penciller { get; set; }
public string Inker { get; set; }
public string Colorist { get; set; }

1215
API/Data/Migrations/20211127200244_MetadataFoundation.Designer.cs generated Normal file
View file

File diff suppressed because it is too large Load diff

203
API/Data/Migrations/20211127200244_MetadataFoundation.cs Normal file
View file

@ -0,0 +1,203 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace API.Data.Migrations
{
public partial class MetadataFoundation : Migration
{
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropColumn(
name: "Summary",
table: "Series");
migrationBuilder.AddColumn<string>(
name: "Summary",
table: "SeriesMetadata",
type: "TEXT",
nullable: true);
migrationBuilder.CreateTable(
name: "ChapterMetadata",
columns: table => new
{
Id = table.Column<int>(type: "INTEGER", nullable: false)
.Annotation("Sqlite:Autoincrement", true),
Title = table.Column<string>(type: "TEXT", nullable: true),
Year = table.Column<string>(type: "TEXT", nullable: true),
StoryArc = table.Column<string>(type: "TEXT", nullable: true),
ChapterId = table.Column<int>(type: "INTEGER", nullable: false)
},
constraints: table =>
{
table.PrimaryKey("PK_ChapterMetadata", x => x.Id);
table.ForeignKey(
name: "FK_ChapterMetadata_Chapter_ChapterId",
column: x => x.ChapterId,
principalTable: "Chapter",
principalColumn: "Id",
onDelete: ReferentialAction.Cascade);
});
migrationBuilder.CreateTable(
name: "Genre",
columns: table => new
{
Id = table.Column<int>(type: "INTEGER", nullable: false)
.Annotation("Sqlite:Autoincrement", true),
Name = table.Column<string>(type: "TEXT", nullable: true),
NormalizedName = table.Column<string>(type: "TEXT", nullable: true)
},
constraints: table =>
{
table.PrimaryKey("PK_Genre", x => x.Id);
});
migrationBuilder.CreateTable(
name: "Person",
columns: table => new
{
Id = table.Column<int>(type: "INTEGER", nullable: false)
.Annotation("Sqlite:Autoincrement", true),
Name = table.Column<string>(type: "TEXT", nullable: true),
NormalizedName = table.Column<string>(type: "TEXT", nullable: true),
Role = table.Column<int>(type: "INTEGER", nullable: false)
},
constraints: table =>
{
table.PrimaryKey("PK_Person", x => x.Id);
});
migrationBuilder.CreateTable(
name: "GenreSeriesMetadata",
columns: table => new
{
GenresId = table.Column<int>(type: "INTEGER", nullable: false),
SeriesMetadatasId = table.Column<int>(type: "INTEGER", nullable: false)
},
constraints: table =>
{
table.PrimaryKey("PK_GenreSeriesMetadata", x => new { x.GenresId, x.SeriesMetadatasId });
table.ForeignKey(
name: "FK_GenreSeriesMetadata_Genre_GenresId",
column: x => x.GenresId,
principalTable: "Genre",
principalColumn: "Id",
onDelete: ReferentialAction.Cascade);
table.ForeignKey(
name: "FK_GenreSeriesMetadata_SeriesMetadata_SeriesMetadatasId",
column: x => x.SeriesMetadatasId,
principalTable: "SeriesMetadata",
principalColumn: "Id",
onDelete: ReferentialAction.Cascade);
});
migrationBuilder.CreateTable(
name: "ChapterMetadataPerson",
columns: table => new
{
ChapterMetadatasId = table.Column<int>(type: "INTEGER", nullable: false),
PeopleId = table.Column<int>(type: "INTEGER", nullable: false)
},
constraints: table =>
{
table.PrimaryKey("PK_ChapterMetadataPerson", x => new { x.ChapterMetadatasId, x.PeopleId });
table.ForeignKey(
name: "FK_ChapterMetadataPerson_ChapterMetadata_ChapterMetadatasId",
column: x => x.ChapterMetadatasId,
principalTable: "ChapterMetadata",
principalColumn: "Id",
onDelete: ReferentialAction.Cascade);
table.ForeignKey(
name: "FK_ChapterMetadataPerson_Person_PeopleId",
column: x => x.PeopleId,
principalTable: "Person",
principalColumn: "Id",
onDelete: ReferentialAction.Cascade);
});
migrationBuilder.CreateTable(
name: "PersonSeriesMetadata",
columns: table => new
{
PeopleId = table.Column<int>(type: "INTEGER", nullable: false),
SeriesMetadatasId = table.Column<int>(type: "INTEGER", nullable: false)
},
constraints: table =>
{
table.PrimaryKey("PK_PersonSeriesMetadata", x => new { x.PeopleId, x.SeriesMetadatasId });
table.ForeignKey(
name: "FK_PersonSeriesMetadata_Person_PeopleId",
column: x => x.PeopleId,
principalTable: "Person",
principalColumn: "Id",
onDelete: ReferentialAction.Cascade);
table.ForeignKey(
name: "FK_PersonSeriesMetadata_SeriesMetadata_SeriesMetadatasId",
column: x => x.SeriesMetadatasId,
principalTable: "SeriesMetadata",
principalColumn: "Id",
onDelete: ReferentialAction.Cascade);
});
migrationBuilder.CreateIndex(
name: "IX_ChapterMetadata_ChapterId",
table: "ChapterMetadata",
column: "ChapterId",
unique: true);
migrationBuilder.CreateIndex(
name: "IX_ChapterMetadataPerson_PeopleId",
table: "ChapterMetadataPerson",
column: "PeopleId");
migrationBuilder.CreateIndex(
name: "IX_Genre_NormalizedName",
table: "Genre",
column: "NormalizedName",
unique: true);
migrationBuilder.CreateIndex(
name: "IX_GenreSeriesMetadata_SeriesMetadatasId",
table: "GenreSeriesMetadata",
column: "SeriesMetadatasId");
migrationBuilder.CreateIndex(
name: "IX_PersonSeriesMetadata_SeriesMetadatasId",
table: "PersonSeriesMetadata",
column: "SeriesMetadatasId");
}
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropTable(
name: "ChapterMetadataPerson");
migrationBuilder.DropTable(
name: "GenreSeriesMetadata");
migrationBuilder.DropTable(
name: "PersonSeriesMetadata");
migrationBuilder.DropTable(
name: "ChapterMetadata");
migrationBuilder.DropTable(
name: "Genre");
migrationBuilder.DropTable(
name: "Person");
migrationBuilder.DropColumn(
name: "Summary",
table: "SeriesMetadata");
migrationBuilder.AddColumn<string>(
name: "Summary",
table: "Series",
type: "TEXT",
nullable: true);
}
}
}

1232
API/Data/Migrations/20211129231007_RemoveChapterMetadata.Designer.cs generated Normal file
View file

File diff suppressed because it is too large Load diff

138
API/Data/Migrations/20211129231007_RemoveChapterMetadata.cs Normal file
View file

@ -0,0 +1,138 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace API.Data.Migrations
{
public partial class RemoveChapterMetadata : Migration
{
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropTable(
name: "ChapterMetadataPerson");
migrationBuilder.DropIndex(
name: "IX_ChapterMetadata_ChapterId",
table: "ChapterMetadata");
migrationBuilder.AddColumn<int>(
name: "ChapterMetadataId",
table: "Person",
type: "INTEGER",
nullable: true);
migrationBuilder.AddColumn<string>(
name: "TitleName",
table: "Chapter",
type: "TEXT",
nullable: true);
migrationBuilder.CreateTable(
name: "ChapterPerson",
columns: table => new
{
ChapterMetadatasId = table.Column<int>(type: "INTEGER", nullable: false),
PeopleId = table.Column<int>(type: "INTEGER", nullable: false)
},
constraints: table =>
{
table.PrimaryKey("PK_ChapterPerson", x => new { x.ChapterMetadatasId, x.PeopleId });
table.ForeignKey(
name: "FK_ChapterPerson_Chapter_ChapterMetadatasId",
column: x => x.ChapterMetadatasId,
principalTable: "Chapter",
principalColumn: "Id",
onDelete: ReferentialAction.Cascade);
table.ForeignKey(
name: "FK_ChapterPerson_Person_PeopleId",
column: x => x.PeopleId,
principalTable: "Person",
principalColumn: "Id",
onDelete: ReferentialAction.Cascade);
});
migrationBuilder.CreateIndex(
name: "IX_Person_ChapterMetadataId",
table: "Person",
column: "ChapterMetadataId");
migrationBuilder.CreateIndex(
name: "IX_ChapterMetadata_ChapterId",
table: "ChapterMetadata",
column: "ChapterId");
migrationBuilder.CreateIndex(
name: "IX_ChapterPerson_PeopleId",
table: "ChapterPerson",
column: "PeopleId");
migrationBuilder.AddForeignKey(
name: "FK_Person_ChapterMetadata_ChapterMetadataId",
table: "Person",
column: "ChapterMetadataId",
principalTable: "ChapterMetadata",
principalColumn: "Id");
}
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropForeignKey(
name: "FK_Person_ChapterMetadata_ChapterMetadataId",
table: "Person");
migrationBuilder.DropTable(
name: "ChapterPerson");
migrationBuilder.DropIndex(
name: "IX_Person_ChapterMetadataId",
table: "Person");
migrationBuilder.DropIndex(
name: "IX_ChapterMetadata_ChapterId",
table: "ChapterMetadata");
migrationBuilder.DropColumn(
name: "ChapterMetadataId",
table: "Person");
migrationBuilder.DropColumn(
name: "TitleName",
table: "Chapter");
migrationBuilder.CreateTable(
name: "ChapterMetadataPerson",
columns: table => new
{
ChapterMetadatasId = table.Column<int>(type: "INTEGER", nullable: false),
PeopleId = table.Column<int>(type: "INTEGER", nullable: false)
},
constraints: table =>
{
table.PrimaryKey("PK_ChapterMetadataPerson", x => new { x.ChapterMetadatasId, x.PeopleId });
table.ForeignKey(
name: "FK_ChapterMetadataPerson_ChapterMetadata_ChapterMetadatasId",
column: x => x.ChapterMetadatasId,
principalTable: "ChapterMetadata",
principalColumn: "Id",
onDelete: ReferentialAction.Cascade);
table.ForeignKey(
name: "FK_ChapterMetadataPerson_Person_PeopleId",
column: x => x.PeopleId,
principalTable: "Person",
principalColumn: "Id",
onDelete: ReferentialAction.Cascade);
});
migrationBuilder.CreateIndex(
name: "IX_ChapterMetadata_ChapterId",
table: "ChapterMetadata",
column: "ChapterId",
unique: true);
migrationBuilder.CreateIndex(
name: "IX_ChapterMetadataPerson_PeopleId",
table: "ChapterMetadataPerson",
column: "PeopleId");
}
}
}

1182
API/Data/Migrations/20211130134642_GenreProvider.Designer.cs generated Normal file
View file

File diff suppressed because it is too large Load diff

86
API/Data/Migrations/20211130134642_GenreProvider.cs Normal file
View file

@ -0,0 +1,86 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace API.Data.Migrations
{
public partial class GenreProvider : Migration
{
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropForeignKey(
name: "FK_Person_ChapterMetadata_ChapterMetadataId",
table: "Person");
migrationBuilder.DropTable(
name: "ChapterMetadata");
migrationBuilder.DropIndex(
name: "IX_Person_ChapterMetadataId",
table: "Person");
migrationBuilder.DropColumn(
name: "ChapterMetadataId",
table: "Person");
migrationBuilder.AddColumn<bool>(
name: "ExternalTag",
table: "Genre",
type: "INTEGER",
nullable: false,
defaultValue: false);
}
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropColumn(
name: "ExternalTag",
table: "Genre");
migrationBuilder.AddColumn<int>(
name: "ChapterMetadataId",
table: "Person",
type: "INTEGER",
nullable: true);
migrationBuilder.CreateTable(
name: "ChapterMetadata",
columns: table => new
{
Id = table.Column<int>(type: "INTEGER", nullable: false)
.Annotation("Sqlite:Autoincrement", true),
ChapterId = table.Column<int>(type: "INTEGER", nullable: false),
StoryArc = table.Column<string>(type: "TEXT", nullable: true),
Title = table.Column<string>(type: "TEXT", nullable: true),
Year = table.Column<string>(type: "TEXT", nullable: true)
},
constraints: table =>
{
table.PrimaryKey("PK_ChapterMetadata", x => x.Id);
table.ForeignKey(
name: "FK_ChapterMetadata_Chapter_ChapterId",
column: x => x.ChapterId,
principalTable: "Chapter",
principalColumn: "Id",
onDelete: ReferentialAction.Cascade);
});
migrationBuilder.CreateIndex(
name: "IX_Person_ChapterMetadataId",
table: "Person",
column: "ChapterMetadataId");
migrationBuilder.CreateIndex(
name: "IX_ChapterMetadata_ChapterId",
table: "ChapterMetadata",
column: "ChapterId");
migrationBuilder.AddForeignKey(
name: "FK_Person_ChapterMetadata_ChapterMetadataId",
table: "Person",
column: "ChapterMetadataId",
principalTable: "ChapterMetadata",
principalColumn: "Id");
}
}
}

1196
API/Data/Migrations/20211201230003_GenreTitle.Designer.cs generated Normal file
View file

File diff suppressed because it is too large Load diff

85
API/Data/Migrations/20211201230003_GenreTitle.cs Normal file
View file

@ -0,0 +1,85 @@
using Microsoft.EntityFrameworkCore.Migrations;
#nullable disable
namespace API.Data.Migrations
{
public partial class GenreTitle : Migration
{
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropIndex(
name: "IX_Genre_NormalizedName",
table: "Genre");
migrationBuilder.RenameColumn(
name: "NormalizedName",
table: "Genre",
newName: "Title");
migrationBuilder.RenameColumn(
name: "Name",
table: "Genre",
newName: "NormalizedTitle");
migrationBuilder.AddColumn<int>(
name: "GenreId",
table: "Chapter",
type: "INTEGER",
nullable: true);
migrationBuilder.CreateIndex(
name: "IX_Genre_NormalizedTitle_ExternalTag",
table: "Genre",
columns: new[] { "NormalizedTitle", "ExternalTag" },
unique: true);
migrationBuilder.CreateIndex(
name: "IX_Chapter_GenreId",
table: "Chapter",
column: "GenreId");
migrationBuilder.AddForeignKey(
name: "FK_Chapter_Genre_GenreId",
table: "Chapter",
column: "GenreId",
principalTable: "Genre",
principalColumn: "Id");
}
protected override void Down(MigrationBuilder migrationBuilder)
{
migrationBuilder.DropForeignKey(
name: "FK_Chapter_Genre_GenreId",
table: "Chapter");
migrationBuilder.DropIndex(
name: "IX_Genre_NormalizedTitle_ExternalTag",
table: "Genre");
migrationBuilder.DropIndex(
name: "IX_Chapter_GenreId",
table: "Chapter");
migrationBuilder.DropColumn(
name: "GenreId",
table: "Chapter");
migrationBuilder.RenameColumn(
name: "Title",
table: "Genre",
newName: "NormalizedName");
migrationBuilder.RenameColumn(
name: "NormalizedTitle",
table: "Genre",
newName: "Name");
migrationBuilder.CreateIndex(
name: "IX_Genre_NormalizedName",
table: "Genre",
column: "NormalizedName",
unique: true);
}
}
}

247
API/Data/Migrations/DataContextModelSnapshot.cs
View file

@ -5,6 +5,8 @@ using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Infrastructure;
using Microsoft.EntityFrameworkCore.Storage.ValueConversion;
#nullable disable
namespace API.Data.Migrations
{
[DbContext(typeof(DataContext))]
@ -13,8 +15,7 @@ namespace API.Data.Migrations
protected override void BuildModel(ModelBuilder modelBuilder)
{
#pragma warning disable 612, 618
modelBuilder
.HasAnnotation("ProductVersion", "5.0.8");
modelBuilder.HasAnnotation("ProductVersion", "6.0.0");
modelBuilder.Entity("API.Entities.AppRole", b =>
{
@ -40,7 +41,7 @@ namespace API.Data.Migrations
.IsUnique()
.HasDatabaseName("RoleNameIndex");
b.ToTable("AspNetRoles");
b.ToTable("AspNetRoles", (string)null);
});
modelBuilder.Entity("API.Entities.AppUser", b =>
@ -118,7 +119,7 @@ namespace API.Data.Migrations
.IsUnique()
.HasDatabaseName("UserNameIndex");
b.ToTable("AspNetUsers");
b.ToTable("AspNetUsers", (string)null);
});
modelBuilder.Entity("API.Entities.AppUserBookmark", b =>
@ -279,7 +280,7 @@ namespace API.Data.Migrations
b.HasIndex("RoleId");
b.ToTable("AspNetUserRoles");
b.ToTable("AspNetUserRoles", (string)null);
});
modelBuilder.Entity("API.Entities.Chapter", b =>
@ -297,6 +298,9 @@ namespace API.Data.Migrations
b.Property<DateTime>("Created")
.HasColumnType("TEXT");
b.Property<int?>("GenreId")
.HasColumnType("INTEGER");
b.Property<bool>("IsSpecial")
.HasColumnType("INTEGER");
@ -315,11 +319,16 @@ namespace API.Data.Migrations
b.Property<string>("Title")
.HasColumnType("TEXT");
b.Property<string>("TitleName")
.HasColumnType("TEXT");
b.Property<int>("VolumeId")
.HasColumnType("INTEGER");
b.HasKey("Id");
b.HasIndex("GenreId");
b.HasIndex("VolumeId");
b.ToTable("Chapter");
@ -382,6 +391,29 @@ namespace API.Data.Migrations
b.ToTable("FolderPath");
});
modelBuilder.Entity("API.Entities.Genre", b =>
{
b.Property<int>("Id")
.ValueGeneratedOnAdd()
.HasColumnType("INTEGER");
b.Property<bool>("ExternalTag")
.HasColumnType("INTEGER");
b.Property<string>("NormalizedTitle")
.HasColumnType("TEXT");
b.Property<string>("Title")
.HasColumnType("TEXT");
b.HasKey("Id");
b.HasIndex("NormalizedTitle", "ExternalTag")
.IsUnique();
b.ToTable("Genre");
});
modelBuilder.Entity("API.Entities.Library", b =>
{
b.Property<int>("Id")
@ -439,6 +471,53 @@ namespace API.Data.Migrations
b.ToTable("MangaFile");
});
modelBuilder.Entity("API.Entities.Metadata.SeriesMetadata", b =>
{
b.Property<int>("Id")
.ValueGeneratedOnAdd()
.HasColumnType("INTEGER");
b.Property<uint>("RowVersion")
.IsConcurrencyToken()
.HasColumnType("INTEGER");
b.Property<int>("SeriesId")
.HasColumnType("INTEGER");
b.Property<string>("Summary")
.HasColumnType("TEXT");
b.HasKey("Id");
b.HasIndex("SeriesId")
.IsUnique();
b.HasIndex("Id", "SeriesId")
.IsUnique();
b.ToTable("SeriesMetadata");
});
modelBuilder.Entity("API.Entities.Person", b =>
{
b.Property<int>("Id")
.ValueGeneratedOnAdd()
.HasColumnType("INTEGER");
b.Property<string>("Name")
.HasColumnType("TEXT");
b.Property<string>("NormalizedName")
.HasColumnType("TEXT");
b.Property<int>("Role")
.HasColumnType("INTEGER");
b.HasKey("Id");
b.ToTable("Person");
});
modelBuilder.Entity("API.Entities.ReadingList", b =>
{
b.Property<int>("Id")
@ -546,9 +625,6 @@ namespace API.Data.Migrations
b.Property<string>("SortName")
.HasColumnType("TEXT");
b.Property<string>("Summary")
.HasColumnType("TEXT");
b.HasKey("Id");
b.HasIndex("LibraryId");
@ -559,30 +635,6 @@ namespace API.Data.Migrations
b.ToTable("Series");
});
modelBuilder.Entity("API.Entities.SeriesMetadata", b =>
{
b.Property<int>("Id")
.ValueGeneratedOnAdd()
.HasColumnType("INTEGER");
b.Property<uint>("RowVersion")
.IsConcurrencyToken()
.HasColumnType("INTEGER");
b.Property<int>("SeriesId")
.HasColumnType("INTEGER");
b.HasKey("Id");
b.HasIndex("SeriesId")
.IsUnique();
b.HasIndex("Id", "SeriesId")
.IsUnique();
b.ToTable("SeriesMetadata");
});
modelBuilder.Entity("API.Entities.ServerSetting", b =>
{
b.Property<int>("Key")
@ -649,6 +701,21 @@ namespace API.Data.Migrations
b.ToTable("AppUserLibrary");
});
modelBuilder.Entity("ChapterPerson", b =>
{
b.Property<int>("ChapterMetadatasId")
.HasColumnType("INTEGER");
b.Property<int>("PeopleId")
.HasColumnType("INTEGER");
b.HasKey("ChapterMetadatasId", "PeopleId");
b.HasIndex("PeopleId");
b.ToTable("ChapterPerson");
});
modelBuilder.Entity("CollectionTagSeriesMetadata", b =>
{
b.Property<int>("CollectionTagsId")
@ -664,6 +731,21 @@ namespace API.Data.Migrations
b.ToTable("CollectionTagSeriesMetadata");
});
modelBuilder.Entity("GenreSeriesMetadata", b =>
{
b.Property<int>("GenresId")
.HasColumnType("INTEGER");
b.Property<int>("SeriesMetadatasId")
.HasColumnType("INTEGER");
b.HasKey("GenresId", "SeriesMetadatasId");
b.HasIndex("SeriesMetadatasId");
b.ToTable("GenreSeriesMetadata");
});
modelBuilder.Entity("Microsoft.AspNetCore.Identity.IdentityRoleClaim<int>", b =>
{
b.Property<int>("Id")
@ -683,7 +765,7 @@ namespace API.Data.Migrations
b.HasIndex("RoleId");
b.ToTable("AspNetRoleClaims");
b.ToTable("AspNetRoleClaims", (string)null);
});
modelBuilder.Entity("Microsoft.AspNetCore.Identity.IdentityUserClaim<int>", b =>
@ -705,7 +787,7 @@ namespace API.Data.Migrations
b.HasIndex("UserId");
b.ToTable("AspNetUserClaims");
b.ToTable("AspNetUserClaims", (string)null);
});
modelBuilder.Entity("Microsoft.AspNetCore.Identity.IdentityUserLogin<int>", b =>
@ -726,7 +808,7 @@ namespace API.Data.Migrations
b.HasIndex("UserId");
b.ToTable("AspNetUserLogins");
b.ToTable("AspNetUserLogins", (string)null);
});
modelBuilder.Entity("Microsoft.AspNetCore.Identity.IdentityUserToken<int>", b =>
@ -745,7 +827,22 @@ namespace API.Data.Migrations
b.HasKey("UserId", "LoginProvider", "Name");
b.ToTable("AspNetUserTokens");
b.ToTable("AspNetUserTokens", (string)null);
});
modelBuilder.Entity("PersonSeriesMetadata", b =>
{
b.Property<int>("PeopleId")
.HasColumnType("INTEGER");
b.Property<int>("SeriesMetadatasId")
.HasColumnType("INTEGER");
b.HasKey("PeopleId", "SeriesMetadatasId");
b.HasIndex("SeriesMetadatasId");
b.ToTable("PersonSeriesMetadata");
});
modelBuilder.Entity("API.Entities.AppUserBookmark", b =>
@ -813,6 +910,10 @@ namespace API.Data.Migrations
modelBuilder.Entity("API.Entities.Chapter", b =>
{
b.HasOne("API.Entities.Genre", null)
.WithMany("Chapters")
.HasForeignKey("GenreId");
b.HasOne("API.Entities.Volume", "Volume")
.WithMany("Chapters")
.HasForeignKey("VolumeId")
@ -844,6 +945,17 @@ namespace API.Data.Migrations
b.Navigation("Chapter");
});
modelBuilder.Entity("API.Entities.Metadata.SeriesMetadata", b =>
{
b.HasOne("API.Entities.Series", "Series")
.WithOne("Metadata")
.HasForeignKey("API.Entities.Metadata.SeriesMetadata", "SeriesId")
.OnDelete(DeleteBehavior.Cascade)
.IsRequired();
b.Navigation("Series");
});
modelBuilder.Entity("API.Entities.ReadingList", b =>
{
b.HasOne("API.Entities.AppUser", "AppUser")
@ -901,17 +1013,6 @@ namespace API.Data.Migrations
b.Navigation("Library");
});
modelBuilder.Entity("API.Entities.SeriesMetadata", b =>
{
b.HasOne("API.Entities.Series", "Series")
.WithOne("Metadata")
.HasForeignKey("API.Entities.SeriesMetadata", "SeriesId")
.OnDelete(DeleteBehavior.Cascade)
.IsRequired();
b.Navigation("Series");
});
modelBuilder.Entity("API.Entities.Volume", b =>
{
b.HasOne("API.Entities.Series", "Series")
@ -938,6 +1039,21 @@ namespace API.Data.Migrations
.IsRequired();
});
modelBuilder.Entity("ChapterPerson", b =>
{
b.HasOne("API.Entities.Chapter", null)
.WithMany()
.HasForeignKey("ChapterMetadatasId")
.OnDelete(DeleteBehavior.Cascade)
.IsRequired();
b.HasOne("API.Entities.Person", null)
.WithMany()
.HasForeignKey("PeopleId")
.OnDelete(DeleteBehavior.Cascade)
.IsRequired();
});
modelBuilder.Entity("CollectionTagSeriesMetadata", b =>
{
b.HasOne("API.Entities.CollectionTag", null)
@ -946,7 +1062,22 @@ namespace API.Data.Migrations
.OnDelete(DeleteBehavior.Cascade)
.IsRequired();
b.HasOne("API.Entities.SeriesMetadata", null)
b.HasOne("API.Entities.Metadata.SeriesMetadata", null)
.WithMany()
.HasForeignKey("SeriesMetadatasId")
.OnDelete(DeleteBehavior.Cascade)
.IsRequired();
});
modelBuilder.Entity("GenreSeriesMetadata", b =>
{
b.HasOne("API.Entities.Genre", null)
.WithMany()
.HasForeignKey("GenresId")
.OnDelete(DeleteBehavior.Cascade)
.IsRequired();
b.HasOne("API.Entities.Metadata.SeriesMetadata", null)
.WithMany()
.HasForeignKey("SeriesMetadatasId")
.OnDelete(DeleteBehavior.Cascade)
@ -989,6 +1120,21 @@ namespace API.Data.Migrations
.IsRequired();
});
modelBuilder.Entity("PersonSeriesMetadata", b =>
{
b.HasOne("API.Entities.Person", null)
.WithMany()
.HasForeignKey("PeopleId")
.OnDelete(DeleteBehavior.Cascade)
.IsRequired();
b.HasOne("API.Entities.Metadata.SeriesMetadata", null)
.WithMany()
.HasForeignKey("SeriesMetadatasId")
.OnDelete(DeleteBehavior.Cascade)
.IsRequired();
});
modelBuilder.Entity("API.Entities.AppRole", b =>
{
b.Navigation("UserRoles");
@ -1014,6 +1160,11 @@ namespace API.Data.Migrations
b.Navigation("Files");
});
modelBuilder.Entity("API.Entities.Genre", b =>
{
b.Navigation("Chapters");
});
modelBuilder.Entity("API.Entities.Library", b =>
{
b.Navigation("Folders");

18
API/Data/Repositories/ChapterRepository.cs
View file

@ -1,4 +1,5 @@
using System.Collections.Generic;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using API.DTOs;
@ -41,7 +42,7 @@ namespace API.Data.Repositories
/// <returns></returns>
public async Task<IChapterInfoDto> GetChapterInfoDtoAsync(int chapterId)
{
return await _context.Chapter
var chapterInfo = await _context.Chapter
.Where(c => c.Id == chapterId)
.Join(_context.Volume, c => c.VolumeId, v => v.Id, (chapter, volume) => new
{
@ -49,8 +50,9 @@ namespace API.Data.Repositories
VolumeNumber = volume.Number,
VolumeId = volume.Id,
chapter.IsSpecial,
chapter.TitleName,
volume.SeriesId,
chapter.Pages
chapter.Pages,
})
.Join(_context.Series, data => data.SeriesId, series => series.Id, (data, series) => new
{
@ -60,11 +62,12 @@ namespace API.Data.Repositories
data.IsSpecial,
data.SeriesId,
data.Pages,
data.TitleName,
SeriesFormat = series.Format,
SeriesName = series.Name,
series.LibraryId
})
.Select(data => new BookInfoDto()
.Select(data => new ChapterInfoDto()
{
ChapterNumber = data.ChapterNumber,
VolumeNumber = data.VolumeNumber + string.Empty,
@ -74,10 +77,13 @@ namespace API.Data.Repositories
SeriesFormat = data.SeriesFormat,
SeriesName = data.SeriesName,
LibraryId = data.LibraryId,
Pages = data.Pages
Pages = data.Pages,
ChapterTitle = data.TitleName
})
.AsNoTracking()
.SingleAsync();
.SingleOrDefaultAsync();
return chapterInfo;
}
public Task<int> GetChapterTotalPagesAsync(int chapterId)

1
API/Data/Repositories/CollectionTagRepository.cs
View file

@ -1,7 +1,6 @@
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using API.DTOs;
using API.DTOs.CollectionTags;
using API.Entities;
using API.Interfaces.Repositories;

35
API/Data/Repositories/FileRepository.cs
View file

@ -1,35 +0,0 @@
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using API.Interfaces.Repositories;
using Microsoft.EntityFrameworkCore;
namespace API.Data.Repositories
{
public class FileRepository : IFileRepository
{
private readonly DataContext _dbContext;
public FileRepository(DataContext context)
{
_dbContext = context;
}
public async Task<IEnumerable<string>> GetFileExtensions()
{
var fileExtensions = await _dbContext.MangaFile
.AsNoTracking()
.Select(x => x.FilePath.ToLower())
.Distinct()
.ToArrayAsync();
var uniqueFileTypes = fileExtensions
.Select(Path.GetExtension)
.Where(x => x is not null)
.Distinct();
return uniqueFileTypes;
}
}
}

56
API/Data/Repositories/GenreRepository.cs Normal file
View file

@ -0,0 +1,56 @@
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using API.Entities;
using API.Interfaces.Repositories;
using AutoMapper;
using Microsoft.EntityFrameworkCore;
namespace API.Data.Repositories;
public class GenreRepository : IGenreRepository
{
private readonly DataContext _context;
private readonly IMapper _mapper;
public GenreRepository(DataContext context, IMapper mapper)
{
_context = context;
_mapper = mapper;
}
public void Attach(Genre genre)
{
_context.Genre.Attach(genre);
}
public void Remove(Genre genre)
{
_context.Genre.Remove(genre);
}
public async Task<Genre> FindByNameAsync(string genreName)
{
var normalizedName = Parser.Parser.Normalize(genreName);
return await _context.Genre
.FirstOrDefaultAsync(g => g.NormalizedTitle.Equals(normalizedName));
}
public async Task RemoveAllGenreNoLongerAssociated(bool removeExternal = false)
{
var genresWithNoConnections = await _context.Genre
.Include(p => p.SeriesMetadatas)
.Include(p => p.Chapters)
.Where(p => p.SeriesMetadatas.Count == 0 && p.Chapters.Count == 0 && p.ExternalTag == removeExternal)
.ToListAsync();
_context.Genre.RemoveRange(genresWithNoConnections);
await _context.SaveChangesAsync();
}
public async Task<IList<Genre>> GetAllGenres()
{
return await _context.Genre.ToListAsync();;
}
}

60
API/Data/Repositories/PersonRepository.cs Normal file
View file

@ -0,0 +1,60 @@
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using API.Entities;
using API.Interfaces.Repositories;
using AutoMapper;
using Microsoft.EntityFrameworkCore;
namespace API.Data.Repositories
{
public class PersonRepository : IPersonRepository
{
private readonly DataContext _context;
private readonly IMapper _mapper;
public PersonRepository(DataContext context, IMapper mapper)
{
_context = context;
_mapper = mapper;
}
public void Attach(Person person)
{
_context.Person.Attach(person);
}
public void Remove(Person person)
{
_context.Person.Remove(person);
}
public async Task<Person> FindByNameAsync(string name)
{
var normalizedName = Parser.Parser.Normalize(name);
return await _context.Person
.Where(p => normalizedName.Equals(p.NormalizedName))
.SingleOrDefaultAsync();
}
public async Task RemoveAllPeopleNoLongerAssociated(bool removeExternal = false)
{
var peopleWithNoConnections = await _context.Person
.Include(p => p.SeriesMetadatas)
.Include(p => p.ChapterMetadatas)
.Where(p => p.SeriesMetadatas.Count == 0 && p.ChapterMetadatas.Count == 0)
.ToListAsync();
_context.Person.RemoveRange(peopleWithNoConnections);
await _context.SaveChangesAsync();
}
public async Task<IList<Person>> GetAllPeople()
{
return await _context.Person
.ToListAsync();
}
}
}

1
API/Data/Repositories/SeriesMetadataRepository.cs
View file

@ -1,4 +1,5 @@
using API.Entities;
using API.Entities.Metadata;
using API.Interfaces.Repositories;
namespace API.Data.Repositories

28
API/Data/Repositories/SeriesRepository.cs
View file

@ -8,6 +8,7 @@ using API.DTOs.CollectionTags;
using API.DTOs.Filtering;
using API.Entities;
using API.Entities.Enums;
using API.Entities.Metadata;
using API.Extensions;
using API.Helpers;
using API.Interfaces.Repositories;
@ -85,6 +86,12 @@ namespace API.Data.Repositories
var query = _context.Series
.Where(s => s.LibraryId == libraryId)
.Include(s => s.Metadata)
.ThenInclude(m => m.People)
.Include(s => s.Metadata)
.ThenInclude(m => m.Genres)
.Include(s => s.Volumes)
.ThenInclude(v => v.Chapters)
.ThenInclude(cm => cm.People)
.Include(s => s.Volumes)
.ThenInclude(v => v.Chapters)
.ThenInclude(c => c.Files)
@ -104,9 +111,15 @@ namespace API.Data.Repositories
return await _context.Series
.Where(s => s.Id == seriesId)
.Include(s => s.Metadata)
.ThenInclude(m => m.People)
.Include(s => s.Metadata)
.ThenInclude(m => m.Genres)
.Include(s => s.Library)
.Include(s => s.Volumes)
.ThenInclude(v => v.Chapters)
.ThenInclude(cm => cm.People)
.Include(s => s.Volumes)
.ThenInclude(v => v.Chapters)
.ThenInclude(c => c.Files)
.AsSplitQuery()
.SingleOrDefaultAsync();
@ -180,6 +193,10 @@ namespace API.Data.Repositories
.Include(s => s.Volumes)
.Include(s => s.Metadata)
.ThenInclude(m => m.CollectionTags)
.Include(s => s.Metadata)
.ThenInclude(m => m.Genres)
.Include(s => s.Metadata)
.ThenInclude(m => m.People)
.Where(s => s.Id == seriesId)
.SingleOrDefaultAsync();
}
@ -374,6 +391,7 @@ namespace API.Data.Repositories
{
var metadataDto = await _context.SeriesMetadata
.Where(metadata => metadata.SeriesId == seriesId)
.Include(m => m.Genres)
.AsNoTracking()
.ProjectTo<SeriesMetadataDto>(_mapper.ConfigurationProvider)
.SingleOrDefaultAsync();
@ -481,17 +499,7 @@ namespace API.Data.Repositories
/// <returns></returns>
private async Task<Tuple<int, int>> GetChunkSize(int libraryId = 0)
{
// TODO: Think about making this bigger depending on number of files a user has in said library
// and number of cores and amount of memory. We can then make an optimal choice
var totalSeries = await GetSeriesCount(libraryId);
// var procCount = Math.Max(Environment.ProcessorCount - 1, 1);
//
// if (totalSeries < procCount * 2 || totalSeries < 50)
// {
// return new Tuple<int, int>(totalSeries, totalSeries);
// }
//
// return new Tuple<int, int>(totalSeries, Math.Max(totalSeries / procCount, 50));
return new Tuple<int, int>(totalSeries, 50);
}

1
API/Data/Repositories/VolumeRepository.cs
View file

@ -157,6 +157,7 @@ namespace API.Data.Repositories
var volumes = await _context.Volume
.Where(vol => vol.SeriesId == seriesId)
.Include(vol => vol.Chapters)
.ThenInclude(c => c.People) // TODO: Measure cost of this
.OrderBy(volume => volume.Number)
.ProjectTo<VolumeDto>(_mapper.ConfigurationProvider)
.AsNoTracking()

3
API/Data/UnitOfWork.cs
View file

@ -31,10 +31,11 @@ namespace API.Data
public IAppUserProgressRepository AppUserProgressRepository => new AppUserProgressRepository(_context);
public ICollectionTagRepository CollectionTagRepository => new CollectionTagRepository(_context, _mapper);
public IFileRepository FileRepository => new FileRepository(_context);
public IChapterRepository ChapterRepository => new ChapterRepository(_context, _mapper);
public IReadingListRepository ReadingListRepository => new ReadingListRepository(_context, _mapper);
public ISeriesMetadataRepository SeriesMetadataRepository => new SeriesMetadataRepository(_context);
public IPersonRepository PersonRepository => new PersonRepository(_context, _mapper);
public IGenreRepository GenreRepository => new GenreRepository(_context, _mapper);
/// <summary>
/// Commits changes to the DB. Completes the open transaction.