profitroll/PhotosAPI
profitroll/PhotosAPI
1
0
Fork 0
Code Issues 4 Pull Requests Releases 7 Wiki Activity

Compare commits

base: profitroll:master
Branches Tags
profitroll:master
profitroll:dev
profitroll:v0.6.0
profitroll:v0.5.0
profitroll:v0.4-fix
profitroll:v0.4
profitroll:v0.3
profitroll:v0.2
profitroll:v0.1
..
compare: profitroll:v0.2
Branches Tags
profitroll:dev
profitroll:master
profitroll:v0.6.0
profitroll:v0.5.0
profitroll:v0.4-fix
profitroll:v0.4
profitroll:v0.3
profitroll:v0.2
profitroll:v0.1

No commits in common. "master" and "v0.2" have entirely different histories.
master ... v0.2

25 changed files with 431 additions and 899 deletions
Show Stats Expand all files Collapse all files

3
.gitignore vendored
View File

@ -153,6 +153,5 @@ cython_debug/
#.idea/ #.idea/
# Custom # Custom
data/ .vscode
.vscode/
config.json config.json

20
.renovaterc
View File

@ -1,20 +0,0 @@
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": [
"config:base"
],
"baseBranches": [
"dev"
],
"packageRules": [
{
"matchUpdateTypes": [
"minor",
"patch",
"pin",
"digest"
],
"automerge": true
}
]
}

20
README.md
View File

@ -1,7 +1,7 @@
<h1 align="center">Photos API</h1> <h1 align="center">Photos API</h1>
<p align="center"> <p align="center">
<a href="https://git.end-play.xyz/profitroll/PhotosAPI/src/branch/master/README.md"><img alt="License: GPL" src="https://img.shields.io/badge/License-GPL-blue"></a> <a href="https://git.end-play.xyz/profitroll/PhotosAPILICENSE"><img alt="License: GPL" src="https://img.shields.io/badge/License-GPL-blue"></a>
<a href="https://git.end-play.xyz/profitroll/PhotosAPI"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a> <a href="https://git.end-play.xyz/profitroll/PhotosAPI"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>
</p> </p>
@ -9,7 +9,7 @@ Small and simple API server for saving photos and videos.
## Dependencies ## Dependencies
* [Python 3.8+](https://www.python.org) (3.9+ recommended) * [Python 3.7+](https://www.python.org) (3.9+ recommended)
* [MongoDB](https://www.mongodb.com) * [MongoDB](https://www.mongodb.com)
* [exiftool](https://exiftool.org) * [exiftool](https://exiftool.org)
* [jpegoptim](https://github.com/tjko/jpegoptim) * [jpegoptim](https://github.com/tjko/jpegoptim)
@ -47,8 +47,7 @@ First you need to have a Python interpreter, MongoDB and optionally git. You can
1. Copy file `config_example.json` to `config.json` 1. Copy file `config_example.json` to `config.json`
2. Open `config.json` using your favorite text editor. For example `nano config.json` 2. Open `config.json` using your favorite text editor. For example `nano config.json`
3. Change `"database"` keys to match your MongoDB setup 3. Change `"database"` keys to match your MongoDB setup
4. Set the key `"secret"` to your JWT secret. You can type in anything, but long secrets are recommended. You can also set environment variable `PHOTOSAPI_SECRET` as an alternative 4. Change `"external_address"` to the ip/http address you may get in responses. By default it's `"localhost"`. This is extremely useful when running behind reverse-proxy.
5. Change `"external_address"` to the ip/http address you may get in responses. By default it's `"localhost"`. This is extremely useful when running behind reverse-proxy.
After configuring everything listed above your API will be able to boot, however further configuration can be done. You can read about it in [repository's wiki](https://git.end-play.xyz/profitroll/PhotosAPI/wiki/Configuration). There's no need to focus on that now, it makes more sense to configure it afterwards. After configuring everything listed above your API will be able to boot, however further configuration can be done. You can read about it in [repository's wiki](https://git.end-play.xyz/profitroll/PhotosAPI/wiki/Configuration). There's no need to focus on that now, it makes more sense to configure it afterwards.
@ -59,19 +58,6 @@ First you need to have a Python interpreter, MongoDB and optionally git. You can
Learn more about available uvicorn arguments using `uvicorn --help` Learn more about available uvicorn arguments using `uvicorn --help`
## Upgrading
When a new version comes out, sometimes you want to upgrade your instance right away. Here's a checklist what to do:
1. Carefully read the patch notes of the version you want to update to and all the versions that came out between the release of your version and the one you want to upgrade to.
Breaking changes will be marked so and config updates will also be described in the patch notes
2. Make a backup of your currently working instance. This includes both the PhotosAPI and the database
3. Download the latest version using git (`git pull` if you cloned the repo in the past) or from the releases
4. Reconfigure the config if needed and apply the changes from the patch notes
5. Upgrade the dependencies in your virtual environment using `pip install -r requirements.txt`
6. Start the migration using `python photos_api.py --migrate` from your virtual environment
7. Test if everything works and troubleshoot/rollback if not
## Using as a service ## Using as a service
It's a good practice to use your API as a systemd service on Linux. Here's a quick overview how that can be done. It's a good practice to use your API as a systemd service on Linux. Here's a quick overview how that can be done.

134
classes/exceptions.py
View File

@ -1,9 +1,7 @@
from typing import Literal from typing import Literal
from fastapi import HTTPException
class AlbumNotFoundError(Exception):
class AlbumNotFoundError(HTTPException):
"""Raises HTTP 404 if no album with this ID found.""" """Raises HTTP 404 if no album with this ID found."""
def __init__(self, id: str): def __init__(self, id: str):
@ -18,7 +16,7 @@ class AlbumNotFoundError(HTTPException):
} }
class AlbumNameNotFoundError(HTTPException): class AlbumNameNotFoundError(Exception):
"""Raises HTTP 404 if no album with this name found.""" """Raises HTTP 404 if no album with this name found."""
def __init__(self, name: str): def __init__(self, name: str):
@ -31,15 +29,9 @@ class AlbumNameNotFoundError(HTTPException):
} }
}, },
} }
super().__init__(
status_code=404,
detail=self.openapi["content"]["application/json"]["example"][
"detail"
].format(name=self.name),
)
class AlbumAlreadyExistsError(HTTPException): class AlbumAlreadyExistsError(Exception):
"""Raises HTTP 409 if album with this name already exists.""" """Raises HTTP 409 if album with this name already exists."""
def __init__(self, name: str): def __init__(self, name: str):
@ -52,15 +44,9 @@ class AlbumAlreadyExistsError(HTTPException):
} }
}, },
} }
super().__init__(
status_code=409,
detail=self.openapi["content"]["application/json"]["example"][
"detail"
].format(name=self.name),
)
class AlbumIncorrectError(HTTPException): class AlbumIncorrectError(Exception):
"""Raises HTTP 406 if album's title or name is invalid.""" """Raises HTTP 406 if album's title or name is invalid."""
def __init__(self, place: Literal["name", "title"], error: str) -> None: def __init__(self, place: Literal["name", "title"], error: str) -> None:
@ -70,19 +56,13 @@ class AlbumIncorrectError(HTTPException):
"description": "Album Name/Title Invalid", "description": "Album Name/Title Invalid",
"content": { "content": {
"application/json": { "application/json": {
"example": {"detail": "Album {place} invalid: {error}"} "example": {"detail": "Album {name/title} invalid: {error}"}
} }
}, },
} }
super().__init__(
status_code=406,
detail=self.openapi["content"]["application/json"]["example"][
"detail"
].format(place=self.place, error=self.error),
)
class PhotoNotFoundError(HTTPException): class PhotoNotFoundError(Exception):
"""Raises HTTP 404 if no photo with this ID found.""" """Raises HTTP 404 if no photo with this ID found."""
def __init__(self, id: str): def __init__(self, id: str):
@ -95,15 +75,9 @@ class PhotoNotFoundError(HTTPException):
} }
}, },
} }
super().__init__(
status_code=404,
detail=self.openapi["content"]["application/json"]["example"][
"detail"
].format(id=self.id),
)
class PhotoSearchQueryEmptyError(HTTPException): class PhotoSearchQueryEmptyError(Exception):
"""Raises HTTP 422 if no photo search query provided.""" """Raises HTTP 422 if no photo search query provided."""
def __init__(self): def __init__(self):
@ -117,13 +91,9 @@ class PhotoSearchQueryEmptyError(HTTPException):
} }
}, },
} }
super().__init__(
status_code=422,
detail=self.openapi["content"]["application/json"]["example"]["detail"],
)
class VideoNotFoundError(HTTPException): class VideoNotFoundError(Exception):
"""Raises HTTP 404 if no video with this ID found.""" """Raises HTTP 404 if no video with this ID found."""
def __init__(self, id: str): def __init__(self, id: str):
@ -136,15 +106,9 @@ class VideoNotFoundError(HTTPException):
} }
}, },
} }
super().__init__(
status_code=404,
detail=self.openapi["content"]["application/json"]["example"][
"detail"
].format(id=self.id),
)
class VideoSearchQueryEmptyError(HTTPException): class VideoSearchQueryEmptyError(Exception):
"""Raises HTTP 422 if no video search query provided.""" """Raises HTTP 422 if no video search query provided."""
def __init__(self): def __init__(self):
@ -158,33 +122,9 @@ class VideoSearchQueryEmptyError(HTTPException):
} }
}, },
} }
super().__init__(
status_code=422,
detail=self.openapi["content"]["application/json"]["example"]["detail"],
)
class SearchLimitInvalidError(HTTPException): class SearchPageInvalidError(Exception):
"""Raises HTTP 400 if search results limit not in valid range."""
def __init__(self):
self.openapi = {
"description": "Invalid Limit",
"content": {
"application/json": {
"example": {
"detail": "Parameter 'limit' must be greater or equal to 1."
}
}
},
}
super().__init__(
status_code=400,
detail=self.openapi["content"]["application/json"]["example"]["detail"],
)
class SearchPageInvalidError(HTTPException):
"""Raises HTTP 400 if page or page size are not in valid range.""" """Raises HTTP 400 if page or page size are not in valid range."""
def __init__(self): def __init__(self):
@ -198,13 +138,9 @@ class SearchPageInvalidError(HTTPException):
} }
}, },
} }
super().__init__(
status_code=400,
detail=self.openapi["content"]["application/json"]["example"]["detail"],
)
class SearchTokenInvalidError(HTTPException): class SearchTokenInvalidError(Exception):
"""Raises HTTP 401 if search token is not valid.""" """Raises HTTP 401 if search token is not valid."""
def __init__(self): def __init__(self):
@ -214,13 +150,9 @@ class SearchTokenInvalidError(HTTPException):
"application/json": {"example": {"detail": "Invalid search token."}} "application/json": {"example": {"detail": "Invalid search token."}}
}, },
} }
super().__init__(
status_code=401,
detail=self.openapi["content"]["application/json"]["example"]["detail"],
)
class UserEmailCodeInvalid(HTTPException): class UserEmailCodeInvalid(Exception):
"""Raises HTTP 400 if email confirmation code is not valid.""" """Raises HTTP 400 if email confirmation code is not valid."""
def __init__(self): def __init__(self):
@ -232,13 +164,9 @@ class UserEmailCodeInvalid(HTTPException):
} }
}, },
} }
super().__init__(
status_code=400,
detail=self.openapi["content"]["application/json"]["example"]["detail"],
)
class UserAlreadyExists(HTTPException): class UserAlreadyExists(Exception):
"""Raises HTTP 409 if user with this name already exists.""" """Raises HTTP 409 if user with this name already exists."""
def __init__(self): def __init__(self):
@ -250,13 +178,9 @@ class UserAlreadyExists(HTTPException):
} }
}, },
} }
super().__init__(
status_code=409,
detail=self.openapi["content"]["application/json"]["example"]["detail"],
)
class AccessTokenInvalidError(HTTPException): class AccessTokenInvalidError(Exception):
"""Raises HTTP 401 if access token is not valid.""" """Raises HTTP 401 if access token is not valid."""
def __init__(self): def __init__(self):
@ -266,13 +190,9 @@ class AccessTokenInvalidError(HTTPException):
"application/json": {"example": {"detail": "Invalid access token."}} "application/json": {"example": {"detail": "Invalid access token."}}
}, },
} }
super().__init__(
status_code=401,
detail=self.openapi["content"]["application/json"]["example"]["detail"],
)
class UserCredentialsInvalid(HTTPException): class UserCredentialsInvalid(Exception):
"""Raises HTTP 401 if user credentials are not valid.""" """Raises HTTP 401 if user credentials are not valid."""
def __init__(self): def __init__(self):
@ -282,27 +202,3 @@ class UserCredentialsInvalid(HTTPException):
"application/json": {"example": {"detail": "Invalid credentials."}} "application/json": {"example": {"detail": "Invalid credentials."}}
}, },
} }
super().__init__(
status_code=401,
detail=self.openapi["content"]["application/json"]["example"]["detail"],
)
class UserMediaQuotaReached(HTTPException):
"""Raises HTTP 403 if user's quota has been reached."""
def __init__(self):
self.openapi = {
"description": "Media Quota Reached",
"content": {
"application/json": {
"example": {
"detail": "Media quota has been reached, media upload impossible."
}
}
},
}
super().__init__(
status_code=403,
detail=self.openapi["content"]["application/json"]["example"]["detail"],
)

9
classes/models.py
View File

@ -1,5 +1,4 @@
from typing import List, Union from typing import List, Union
from pydantic import BaseModel from pydantic import BaseModel
@ -72,11 +71,3 @@ class SearchResultsPhoto(BaseModel):
class SearchResultsVideo(BaseModel): class SearchResultsVideo(BaseModel):
results: List[VideoSearch] results: List[VideoSearch]
next_page: Union[str, None] next_page: Union[str, None]
class RandomSearchResultsPhoto(BaseModel):
results: List[PhotoSearch]
class RandomSearchResultsVideo(BaseModel):
results: List[VideoSearch]

2
config_example.json
View File

@ -6,7 +6,6 @@
"user": null, "user": null,
"password": null "password": null
}, },
"secret": "",
"messages": { "messages": {
"email_confirmed": "Email confirmed. You can now log in." "email_confirmed": "Email confirmed. You can now log in."
}, },
@ -15,7 +14,6 @@
"media_token_valid_hours": 12, "media_token_valid_hours": 12,
"registration_enabled": true, "registration_enabled": true,
"registration_requires_confirmation": false, "registration_requires_confirmation": false,
"default_user_quota": 10000,
"mailer": { "mailer": {
"smtp": { "smtp": {
"host": "", "host": "",

68
extensions/albums.py
View File

@ -1,6 +1,5 @@
import re import re
from os import makedirs, rename from os import makedirs, path, rename
from pathlib import Path
from shutil import rmtree from shutil import rmtree
from typing import Union from typing import Union
@ -47,12 +46,14 @@ async def album_create(
if 2 > len(title) > 40: if 2 > len(title) > 40:
raise AlbumIncorrectError("title", "must be >2 and <40 characters.") raise AlbumIncorrectError("title", "must be >2 and <40 characters.")
if (await col_albums.find_one({"name": name})) is not None: if col_albums.find_one({"name": name}) is not None:
raise AlbumAlreadyExistsError(name) raise AlbumAlreadyExistsError(name)
makedirs(Path(f"data/users/{current_user.user}/albums/{name}"), exist_ok=True) makedirs(
path.join("data", "users", current_user.user, "albums", name), exist_ok=True
)
uploaded = await col_albums.insert_one( uploaded = col_albums.insert_one(
{"user": current_user.user, "name": name, "title": title, "cover": None} {"user": current_user.user, "name": name, "title": title, "cover": None}
) )
@ -67,10 +68,9 @@ async def album_find(
current_user: User = Security(get_current_active_user, scopes=["albums.list"]), current_user: User = Security(get_current_active_user, scopes=["albums.list"]),
): ):
output = {"results": []} output = {"results": []}
albums = list(col_albums.find({"user": current_user.user, "name": re.compile(q)}))
async for album in col_albums.find( for album in albums:
{"user": current_user.user, "name": re.compile(q)}
):
output["results"].append( output["results"].append(
{ {
"id": album["_id"].__str__(), "id": album["_id"].__str__(),
@ -103,17 +103,17 @@ async def album_patch(
current_user: User = Security(get_current_active_user, scopes=["albums.write"]), current_user: User = Security(get_current_active_user, scopes=["albums.write"]),
): ):
try: try:
album = await col_albums.find_one({"_id": ObjectId(id)}) album = col_albums.find_one({"_id": ObjectId(id)})
if album is None: if album is None:
raise InvalidId(id) raise InvalidId(id)
except InvalidId as exc: except InvalidId:
raise AlbumNotFoundError(id) from exc raise AlbumNotFoundError(id)
if title is None: if title is not None:
title = album["title"] if 2 > len(title) > 40:
elif 2 > len(title) > 40:
raise AlbumIncorrectError("title", "must be >2 and <40 characters.") raise AlbumIncorrectError("title", "must be >2 and <40 characters.")
else:
title = album["title"]
if name is not None: if name is not None:
if re.search(re.compile("^[a-z,0-9,_]*$"), name) is False: if re.search(re.compile("^[a-z,0-9,_]*$"), name) is False:
@ -123,10 +123,10 @@ async def album_patch(
if 2 > len(name) > 20: if 2 > len(name) > 20:
raise AlbumIncorrectError("name", "must be >2 and <20 characters.") raise AlbumIncorrectError("name", "must be >2 and <20 characters.")
rename( rename(
Path(f"data/users/{current_user.user}/albums/{album['name']}"), path.join("data", "users", current_user.user, "albums", album["name"]),
Path(f"data/users/{current_user.user}/albums/{name}"), path.join("data", "users", current_user.user, "albums", name),
) )
await col_photos.update_many( col_photos.update_many(
{"user": current_user.user, "album": album["name"]}, {"user": current_user.user, "album": album["name"]},
{"$set": {"album": name}}, {"$set": {"album": name}},
) )
@ -134,14 +134,12 @@ async def album_patch(
name = album["name"] name = album["name"]
if cover is not None: if cover is not None:
image = await col_photos.find_one( image = col_photos.find_one({"_id": ObjectId(cover), "album": album["name"]})
{"_id": ObjectId(cover), "album": album["name"]}
)
cover = image["_id"].__str__() if image is not None else album["cover"] cover = image["_id"].__str__() if image is not None else album["cover"]
else: else:
cover = album["cover"] cover = album["cover"]
await col_albums.update_one( col_albums.update_one(
{"_id": ObjectId(id)}, {"$set": {"name": name, "title": title, "cover": cover}} {"_id": ObjectId(id)}, {"$set": {"name": name, "title": title, "cover": cover}}
) )
@ -169,11 +167,11 @@ async def album_put(
current_user: User = Security(get_current_active_user, scopes=["albums.write"]), current_user: User = Security(get_current_active_user, scopes=["albums.write"]),
): ):
try: try:
album = await col_albums.find_one({"_id": ObjectId(id)}) album = col_albums.find_one({"_id": ObjectId(id)})
if album is None: if album is None:
raise InvalidId(id) raise InvalidId(id)
except InvalidId as exc: except InvalidId:
raise AlbumNotFoundError(id) from exc raise AlbumNotFoundError(id)
if re.search(re.compile("^[a-z,0-9,_]*$"), name) is False: if re.search(re.compile("^[a-z,0-9,_]*$"), name) is False:
raise AlbumIncorrectError("name", "can only contain a-z, 0-9 and _ characters.") raise AlbumIncorrectError("name", "can only contain a-z, 0-9 and _ characters.")
@ -184,18 +182,18 @@ async def album_put(
if 2 > len(title) > 40: if 2 > len(title) > 40:
raise AlbumIncorrectError("title", "must be >2 and <40 characters.") raise AlbumIncorrectError("title", "must be >2 and <40 characters.")
image = await col_photos.find_one({"_id": ObjectId(cover), "album": album["name"]}) image = col_photos.find_one({"_id": ObjectId(cover), "album": album["name"]})
cover = image["_id"].__str__() if image is not None else None # type: ignore cover = image["_id"].__str__() if image is not None else None # type: ignore
rename( rename(
Path(f"data/users/{current_user.user}/albums/{album['name']}"), path.join("data", "users", current_user.user, "albums", album["name"]),
Path(f"data/users/{current_user.user}/albums/{name}"), path.join("data", "users", current_user.user, "albums", name),
) )
await col_photos.update_many( col_photos.update_many(
{"user": current_user.user, "album": album["name"]}, {"$set": {"album": name}} {"user": current_user.user, "album": album["name"]}, {"$set": {"album": name}}
) )
await col_albums.update_one( col_albums.update_one(
{"_id": ObjectId(id)}, {"$set": {"name": name, "title": title, "cover": cover}} {"_id": ObjectId(id)}, {"$set": {"name": name, "title": title, "cover": cover}}
) )
@ -216,14 +214,14 @@ async def album_delete(
current_user: User = Security(get_current_active_user, scopes=["albums.write"]), current_user: User = Security(get_current_active_user, scopes=["albums.write"]),
): ):
try: try:
album = await col_albums.find_one_and_delete({"_id": ObjectId(id)}) album = col_albums.find_one_and_delete({"_id": ObjectId(id)})
if album is None: if album is None:
raise InvalidId(id) raise InvalidId(id)
except InvalidId as exc: except InvalidId:
raise AlbumNotFoundError(id) from exc raise AlbumNotFoundError(id)
await col_photos.delete_many({"album": album["name"]}) col_photos.delete_many({"album": album["name"]})
rmtree(Path(f"data/users/{current_user.user}/albums/{album['name']}")) rmtree(path.join("data", "users", current_user.user, "albums", album["name"]))
return Response(status_code=HTTP_204_NO_CONTENT) return Response(status_code=HTTP_204_NO_CONTENT)

48
extensions/exceptions.py
View File

@ -1,34 +1,16 @@
from fastapi import Request from fastapi import Request
from fastapi.responses import UJSONResponse from fastapi.responses import UJSONResponse
from modules.app import app
from classes.exceptions import *
from starlette.status import ( from starlette.status import (
HTTP_400_BAD_REQUEST, HTTP_400_BAD_REQUEST,
HTTP_401_UNAUTHORIZED, HTTP_401_UNAUTHORIZED,
HTTP_403_FORBIDDEN,
HTTP_404_NOT_FOUND, HTTP_404_NOT_FOUND,
HTTP_406_NOT_ACCEPTABLE, HTTP_406_NOT_ACCEPTABLE,
HTTP_409_CONFLICT, HTTP_409_CONFLICT,
HTTP_422_UNPROCESSABLE_ENTITY, HTTP_422_UNPROCESSABLE_ENTITY,
) )
from classes.exceptions import (
AccessTokenInvalidError,
AlbumAlreadyExistsError,
AlbumIncorrectError,
AlbumNotFoundError,
PhotoNotFoundError,
PhotoSearchQueryEmptyError,
SearchLimitInvalidError,
SearchPageInvalidError,
SearchTokenInvalidError,
UserAlreadyExists,
UserCredentialsInvalid,
UserEmailCodeInvalid,
UserMediaQuotaReached,
VideoNotFoundError,
VideoSearchQueryEmptyError,
)
from modules.app import app
@app.exception_handler(AlbumNotFoundError) @app.exception_handler(AlbumNotFoundError)
async def album_not_found_exception_handler(request: Request, exc: AlbumNotFoundError): async def album_not_found_exception_handler(request: Request, exc: AlbumNotFoundError):
@ -96,24 +78,12 @@ async def video_search_query_empty_exception_handler(
) )
@app.exception_handler(SearchLimitInvalidError)
async def search_limit_invalid_exception_handler(
request: Request, exc: SearchLimitInvalidError
):
return UJSONResponse(
status_code=HTTP_400_BAD_REQUEST,
content={
"detail": "Parameter 'limit' must be greater or equal to 1."
},
)
@app.exception_handler(SearchPageInvalidError) @app.exception_handler(SearchPageInvalidError)
async def search_page_invalid_exception_handler( async def search_page_invalid_exception_handler(
request: Request, exc: SearchPageInvalidError request: Request, exc: SearchPageInvalidError
): ):
return UJSONResponse( return UJSONResponse(
status_code=HTTP_401_UNAUTHORIZED, status_code=HTTP_400_BAD_REQUEST,
content={ content={
"detail": "Parameters 'page' and 'page_size' must be greater or equal to 1." "detail": "Parameters 'page' and 'page_size' must be greater or equal to 1."
}, },
@ -127,7 +97,7 @@ async def search_token_invalid_exception_handler(
return UJSONResponse( return UJSONResponse(
status_code=HTTP_401_UNAUTHORIZED, status_code=HTTP_401_UNAUTHORIZED,
content={ content={
"detail": "Invalid search token." "detail": "Parameters 'page' and 'page_size' must be greater or equal to 1."
}, },
) )
@ -170,13 +140,3 @@ async def user_credentials_invalid_exception_handler(
status_code=HTTP_401_UNAUTHORIZED, status_code=HTTP_401_UNAUTHORIZED,
content={"detail": "Invalid credentials."}, content={"detail": "Invalid credentials."},
) )
@app.exception_handler(UserMediaQuotaReached)
async def user_media_quota_reached_exception_handler(
request: Request, exc: UserMediaQuotaReached
):
return UJSONResponse(
status_code=HTTP_403_FORBIDDEN,
content={"detail": "Media quota has been reached, media upload impossible."},
)

25
extensions/pages.py
View File

@ -1,36 +1,31 @@
from pathlib import Path from os import path
import aiofiles
from fastapi.responses import HTMLResponse, Response
from modules.app import app from modules.app import app
from fastapi.responses import HTMLResponse, Response
@app.get("/pages/matter.css", include_in_schema=False) @app.get("/pages/matter.css", include_in_schema=False)
async def page_matter(): async def page_matter():
async with aiofiles.open(Path("pages/matter.css"), "r", encoding="utf-8") as f: with open(path.join("pages", "matter.css"), "r", encoding="utf-8") as f:
output = await f.read() output = f.read()
return Response(content=output) return Response(content=output)
@app.get("/pages/{page}/{file}", include_in_schema=False) @app.get("/pages/{page}/{file}", include_in_schema=False)
async def page_assets(page: str, file: str): async def page_assets(page: str, file: str):
async with aiofiles.open(Path(f"pages/{page}/{file}"), "r", encoding="utf-8") as f: with open(path.join("pages", page, file), "r", encoding="utf-8") as f:
output = await f.read() output = f.read()
return Response(content=output) return Response(content=output)
@app.get("/", include_in_schema=False) @app.get("/", include_in_schema=False)
async def page_home(): async def page_home():
async with aiofiles.open(Path("pages/home/index.html"), "r", encoding="utf-8") as f: with open(path.join("pages", "home", "index.html"), "r", encoding="utf-8") as f:
output = await f.read() output = f.read()
return HTMLResponse(content=output) return HTMLResponse(content=output)
@app.get("/register", include_in_schema=False) @app.get("/register", include_in_schema=False)
async def page_register(): async def page_register():
async with aiofiles.open( with open(path.join("pages", "register", "index.html"), "r", encoding="utf-8") as f:
Path("pages/register/index.html"), "r", encoding="utf-8" output = f.read()
) as f:
output = await f.read()
return HTMLResponse(content=output) return HTMLResponse(content=output)

321
extensions/photos.py
View File

@ -1,47 +1,26 @@
import logging
import re import re
from datetime import datetime, timedelta, timezone import pickle
from os import makedirs, path, remove, system
from pathlib import Path
from random import randint
from secrets import token_urlsafe from secrets import token_urlsafe
from shutil import move from shutil import move
from threading import Thread from threading import Thread
from typing import Union from typing import Union
from uuid import uuid4 from uuid import uuid4
import aiofiles
from bson.errors import InvalidId
from bson.objectid import ObjectId
from fastapi import Security, UploadFile
from fastapi.responses import Response, UJSONResponse
from jose import JWTError, jwt
from magic import Magic from magic import Magic
from plum.exceptions import UnpackError from datetime import datetime, timedelta, timezone
from pydantic import ValidationError from os import makedirs, path, remove, system
from pymongo import DESCENDING
from starlette.status import HTTP_204_NO_CONTENT, HTTP_409_CONFLICT
from pydantic import ValidationError
from classes.exceptions import ( from classes.exceptions import (
AccessTokenInvalidError, AccessTokenInvalidError,
AlbumNameNotFoundError, AlbumNameNotFoundError,
PhotoNotFoundError, PhotoNotFoundError,
PhotoSearchQueryEmptyError, PhotoSearchQueryEmptyError,
SearchLimitInvalidError,
SearchPageInvalidError, SearchPageInvalidError,
SearchTokenInvalidError, SearchTokenInvalidError,
UserMediaQuotaReached,
) )
from classes.models import ( from classes.models import Photo, PhotoPublic, SearchResultsPhoto
Photo,
PhotoPublic,
RandomSearchResultsPhoto,
SearchResultsPhoto,
)
from modules.app import app
from modules.database import col_albums, col_photos, col_tokens, col_videos
from modules.exif_reader import extract_location from modules.exif_reader import extract_location
from modules.hasher import get_duplicates, get_phash from modules.hasher import get_phash, get_duplicates
from modules.scheduler import scheduler from modules.scheduler import scheduler
from modules.security import ( from modules.security import (
ALGORITHM, ALGORITHM,
@ -52,18 +31,31 @@ from modules.security import (
get_current_active_user, get_current_active_user,
get_user, get_user,
) )
from modules.utils import configGet from modules.app import app
from modules.database import col_photos, col_albums, col_tokens
from pymongo import DESCENDING
from bson.objectid import ObjectId
from bson.errors import InvalidId
from plum.exceptions import UnpackError
from jose import JWTError, jwt
logger = logging.getLogger(__name__) from fastapi import UploadFile, Security
from fastapi.responses import UJSONResponse, Response
from fastapi.exceptions import HTTPException
from starlette.status import (
HTTP_204_NO_CONTENT,
HTTP_401_UNAUTHORIZED,
HTTP_409_CONFLICT,
)
from modules.utils import configGet, logWrite
async def compress_image(image_path: str): async def compress_image(image_path: str):
image_type = Magic(mime=True).from_file(image_path) image_type = Magic(mime=True).from_file(image_path)
if image_type not in ["image/jpeg", "image/png"]: if image_type not in ["image/jpeg", "image/png"]:
logger.info( logWrite(f"Not compressing {image_path} because its mime is '{image_type}'")
"Not compressing %s because its mime is '%s'", image_path, image_type
)
return return
size_before = path.getsize(image_path) / 1024 size_before = path.getsize(image_path) / 1024
@ -79,20 +71,16 @@ async def compress_image(image_path: str):
return return
task.start() task.start()
logger.info("Compressing '%s'...", Path(image_path).name) logWrite(f"Compressing '{path.split(image_path)[-1]}'...")
task.join() task.join()
size_after = path.getsize(image_path) / 1024 size_after = path.getsize(image_path) / 1024
logger.info( logWrite(
"Compressed '%s' from %s Kb to %s Kb", f"Compressed '{path.split(image_path)[-1]}' from {size_before} Kb to {size_after} Kb"
Path(image_path).name,
size_before,
size_after,
) )
photo_post_responses = { photo_post_responses = {
403: UserMediaQuotaReached().openapi,
404: AlbumNameNotFoundError("name").openapi, 404: AlbumNameNotFoundError("name").openapi,
409: { 409: {
"description": "Image Duplicates Found", "description": "Image Duplicates Found",
@ -124,40 +112,39 @@ async def photo_upload(
caption: Union[str, None] = None, caption: Union[str, None] = None,
current_user: User = Security(get_current_active_user, scopes=["photos.write"]), current_user: User = Security(get_current_active_user, scopes=["photos.write"]),
): ):
if (await col_albums.find_one({"user": current_user.user, "name": album})) is None: if col_albums.find_one({"user": current_user.user, "name": album}) is None:
raise AlbumNameNotFoundError(album) raise AlbumNameNotFoundError(album)
user_media_count = ( makedirs(
await col_photos.count_documents({"user": current_user.user}) path.join("data", "users", current_user.user, "albums", album), exist_ok=True
) + (await col_videos.count_documents({"user": current_user.user})) )
if user_media_count >= current_user.quota and not current_user.quota == -1: # type: ignore
raise UserMediaQuotaReached()
makedirs(Path(f"data/users/{current_user.user}/albums/{album}"), exist_ok=True)
filename = file.filename filename = file.filename
if Path(f"data/users/{current_user.user}/albums/{album}/{file.filename}").exists(): if path.exists(
path.join("data", "users", current_user.user, "albums", album, file.filename)
):
base_name = file.filename.split(".")[:-1] base_name = file.filename.split(".")[:-1]
extension = file.filename.split(".")[-1] extension = file.filename.split(".")[-1]
filename = ( filename = (
".".join(base_name) + f"_{int(datetime.now().timestamp())}." + extension ".".join(base_name) + f"_{int(datetime.now().timestamp())}." + extension
) )
async with aiofiles.open( with open(
Path(f"data/users/{current_user.user}/albums/{album}/{filename}"), "wb" path.join("data", "users", current_user.user, "albums", album, filename), "wb"
) as f: ) as f:
await f.write(await file.read()) f.write(await file.read())
file_hash = await get_phash( file_hash = await get_phash(
Path(f"data/users/{current_user.user}/albums/{album}/{filename}") path.join("data", "users", current_user.user, "albums", album, filename)
) )
duplicates = await get_duplicates(file_hash, album) duplicates = await get_duplicates(file_hash, album)
if len(duplicates) > 0 and not ignore_duplicates: if len(duplicates) > 0 and ignore_duplicates is False:
if configGet("media_token_access") is True: if configGet("media_token_access") is True:
duplicates_ids = [entry["id"] for entry in duplicates] duplicates_ids = []
for entry in duplicates:
duplicates_ids.append(entry["id"])
access_token = create_access_token( access_token = create_access_token(
data={ data={
"sub": current_user.user, "sub": current_user.user,
@ -167,7 +154,7 @@ async def photo_upload(
expires_delta=timedelta(hours=configGet("media_token_valid_hours")), expires_delta=timedelta(hours=configGet("media_token_valid_hours")),
) )
access_token_short = uuid4().hex[:12].lower() access_token_short = uuid4().hex[:12].lower()
await col_tokens.insert_one( col_tokens.insert_one(
{ {
"short": access_token_short, "short": access_token_short,
"access_token": access_token, "access_token": access_token,
@ -187,12 +174,12 @@ async def photo_upload(
try: try:
coords = extract_location( coords = extract_location(
Path(f"data/users/{current_user.user}/albums/{album}/{filename}") path.join("data", "users", current_user.user, "albums", album, filename)
) )
except (UnpackError, ValueError): except (UnpackError, ValueError):
coords = {"lng": 0.0, "lat": 0.0, "alt": 0.0} coords = {"lng": 0.0, "lat": 0.0, "alt": 0.0}
uploaded = await col_photos.insert_one( uploaded = col_photos.insert_one(
{ {
"user": current_user.user, "user": current_user.user,
"album": album, "album": album,
@ -207,12 +194,14 @@ async def photo_upload(
} }
) )
if compress: if compress is True:
scheduler.add_job( scheduler.add_job(
compress_image, compress_image,
trigger="date", trigger="date",
run_date=datetime.now() + timedelta(seconds=1), run_date=datetime.now() + timedelta(seconds=1),
args=[Path(f"data/users/{current_user.user}/albums/{album}/{filename}")], args=[
path.join("data", "users", current_user.user, "albums", album, filename)
],
) )
return UJSONResponse( return UJSONResponse(
@ -240,7 +229,7 @@ if configGet("media_token_access") is True:
responses=photo_get_token_responses, responses=photo_get_token_responses,
) )
async def photo_get_token(token: str, id: int): async def photo_get_token(token: str, id: int):
db_entry = await col_tokens.find_one({"short": token}) db_entry = col_tokens.find_one({"short": token})
if db_entry is None: if db_entry is None:
raise AccessTokenInvalidError() raise AccessTokenInvalidError()
@ -255,74 +244,57 @@ if configGet("media_token_access") is True:
raise AccessTokenInvalidError() raise AccessTokenInvalidError()
token_scopes = payload.get("scopes", []) token_scopes = payload.get("scopes", [])
token_data = TokenData(scopes=token_scopes, user=user) token_data = TokenData(scopes=token_scopes, user=user)
except (JWTError, ValidationError) as exc: except (JWTError, ValidationError) as exp:
raise AccessTokenInvalidError() from exc print(exp, flush=True)
raise AccessTokenInvalidError()
user_record = await get_user(user=token_data.user) user = get_user(user=token_data.user)
if id not in payload.get("allowed", []): if id not in payload.get("allowed", []):
raise AccessTokenInvalidError() raise AccessTokenInvalidError()
try: try:
image = await col_photos.find_one({"_id": ObjectId(id)}) image = col_photos.find_one({"_id": ObjectId(id)})
if image is None: if image is None:
raise InvalidId(id) raise InvalidId(id)
except InvalidId as exc: except InvalidId:
raise PhotoNotFoundError(id) from exc raise PhotoNotFoundError(id)
image_path = Path( image_path = path.join(
f"data/users/{user_record.user}/albums/{image['album']}/{image['filename']}" "data", "users", user.user, "albums", image["album"], image["filename"]
) )
mime = Magic(mime=True).from_file(image_path) mime = Magic(mime=True).from_file(image_path)
async with aiofiles.open(image_path, "rb") as f: with open(image_path, "rb") as f:
image_file = await f.read() image_file = f.read()
return Response(image_file, media_type=mime) return Response(image_file, media_type=mime)
photo_get_responses = { photo_get_responses = {404: PhotoNotFoundError("id").openapi}
200: {
"content": {
"application/octet-stream": {
"schema": {
"type": "string",
"format": "binary",
"contentMediaType": "image/*",
}
}
}
},
404: PhotoNotFoundError("id").openapi,
}
@app.get( @app.get("/photos/{id}", description="Get a photo by id", responses=photo_get_responses)
"/photos/{id}",
description="Get a photo by id",
responses=photo_get_responses,
response_class=Response,
)
async def photo_get( async def photo_get(
id: str, id: str,
current_user: User = Security(get_current_active_user, scopes=["photos.read"]), current_user: User = Security(get_current_active_user, scopes=["photos.read"]),
): ):
try: try:
image = await col_photos.find_one({"_id": ObjectId(id)}) image = col_photos.find_one({"_id": ObjectId(id)})
if image is None: if image is None:
raise InvalidId(id) raise InvalidId(id)
except InvalidId as exc: except InvalidId:
raise PhotoNotFoundError(id) from exc raise PhotoNotFoundError(id)
image_path = Path( image_path = path.join(
f"data/users/{current_user.user}/albums/{image['album']}/{image['filename']}" "data", "users", current_user.user, "albums", image["album"], image["filename"]
) )
mime = Magic(mime=True).from_file(image_path) mime = Magic(mime=True).from_file(image_path)
async with aiofiles.open(image_path, "rb") as f: with open(image_path, "rb") as f:
image_file = await f.read() image_file = f.read()
return Response(image_file, media_type=mime) return Response(image_file, media_type=mime)
@ -342,18 +314,20 @@ async def photo_move(
current_user: User = Security(get_current_active_user, scopes=["photos.write"]), current_user: User = Security(get_current_active_user, scopes=["photos.write"]),
): ):
try: try:
image = await col_photos.find_one({"_id": ObjectId(id)}) image = col_photos.find_one({"_id": ObjectId(id)})
if image is None: if image is None:
raise InvalidId(id) raise InvalidId(id)
except InvalidId as exc: except InvalidId:
raise PhotoNotFoundError(id) from exc raise PhotoNotFoundError(id)
if (await col_albums.find_one({"user": current_user.user, "name": album})) is None: if col_albums.find_one({"user": current_user.user, "name": album}) is None:
raise AlbumNameNotFoundError(album) raise AlbumNameNotFoundError(album)
if Path( if path.exists(
f"data/users/{current_user.user}/albums/{album}/{image['filename']}" path.join(
).exists(): "data", "users", current_user.user, "albums", album, image["filename"]
)
):
base_name = image["filename"].split(".")[:-1] base_name = image["filename"].split(".")[:-1]
extension = image["filename"].split(".")[-1] extension = image["filename"].split(".")[-1]
filename = ( filename = (
@ -362,7 +336,7 @@ async def photo_move(
else: else:
filename = image["filename"] filename = image["filename"]
await col_photos.find_one_and_update( col_photos.find_one_and_update(
{"_id": ObjectId(id)}, {"_id": ObjectId(id)},
{ {
"$set": { "$set": {
@ -374,10 +348,15 @@ async def photo_move(
) )
move( move(
Path( path.join(
f"data/users/{current_user.user}/albums/{image['album']}/{image['filename']}" "data",
"users",
current_user.user,
"albums",
image["album"],
image["filename"],
), ),
Path(f"data/users/{current_user.user}/albums/{album}/{filename}"), path.join("data", "users", current_user.user, "albums", album, filename),
) )
return UJSONResponse( return UJSONResponse(
@ -404,13 +383,13 @@ async def photo_patch(
current_user: User = Security(get_current_active_user, scopes=["photos.write"]), current_user: User = Security(get_current_active_user, scopes=["photos.write"]),
): ):
try: try:
image = await col_photos.find_one({"_id": ObjectId(id)}) image = col_photos.find_one({"_id": ObjectId(id)})
if image is None: if image is None:
raise InvalidId(id) raise InvalidId(id)
except InvalidId as exc: except InvalidId:
raise PhotoNotFoundError(id) from exc raise PhotoNotFoundError(id)
await col_photos.find_one_and_update( col_photos.find_one_and_update(
{"_id": ObjectId(id)}, {"_id": ObjectId(id)},
{"$set": {"caption": caption, "dates.modified": datetime.now(tz=timezone.utc)}}, {"$set": {"caption": caption, "dates.modified": datetime.now(tz=timezone.utc)}},
) )
@ -438,87 +417,31 @@ async def photo_delete(
current_user: User = Security(get_current_active_user, scopes=["photos.write"]), current_user: User = Security(get_current_active_user, scopes=["photos.write"]),
): ):
try: try:
image = await col_photos.find_one_and_delete({"_id": ObjectId(id)}) image = col_photos.find_one_and_delete({"_id": ObjectId(id)})
if image is None: if image is None:
raise InvalidId(id) raise InvalidId(id)
except InvalidId as exc: except InvalidId:
raise PhotoNotFoundError(id) from exc raise PhotoNotFoundError(id)
album = await col_albums.find_one({"name": image["album"]}) album = col_albums.find_one({"name": image["album"]})
if album is not None and album["cover"] == image["_id"].__str__(): if album is not None and album["cover"] == image["_id"].__str__():
await col_albums.update_one({"name": image["album"]}, {"$set": {"cover": None}}) col_albums.update_one({"name": image["album"]}, {"$set": {"cover": None}})
remove( remove(
Path( path.join(
f"data/users/{current_user.user}/albums/{image['album']}/{image['filename']}" "data",
"users",
current_user.user,
"albums",
image["album"],
image["filename"],
) )
) )
return Response(status_code=HTTP_204_NO_CONTENT) return Response(status_code=HTTP_204_NO_CONTENT)
photo_random_responses = {
400: SearchLimitInvalidError().openapi,
404: AlbumNameNotFoundError("name").openapi,
}
@app.get(
"/albums/{album}/photos/random",
description="Get one random photo, optionally by caption",
response_class=UJSONResponse,
response_model=RandomSearchResultsPhoto,
responses=photo_random_responses,
)
async def photo_random(
album: str,
caption: Union[str, None] = None,
limit: int = 100,
current_user: User = Security(get_current_active_user, scopes=["photos.list"]),
):
if (await col_albums.find_one({"user": current_user.user, "name": album})) is None:
raise AlbumNameNotFoundError(album)
if limit <= 0:
raise SearchLimitInvalidError()
output = {"results": []}
db_query = (
{
"user": current_user.user,
"album": album,
"caption": re.compile(caption),
}
if caption is not None
else {
"user": current_user.user,
"album": album,
}
)
documents_count = await col_photos.count_documents(db_query)
skip = randint(0, documents_count - 1) if documents_count > 1 else 0
async for image in col_photos.aggregate(
[
{"$match": db_query},
{"$skip": skip},
{"$limit": limit},
]
):
output["results"].append(
{
"id": image["_id"].__str__(),
"filename": image["filename"],
"caption": image["caption"],
}
)
return UJSONResponse(output)
photo_find_responses = { photo_find_responses = {
400: SearchPageInvalidError().openapi, 400: SearchPageInvalidError().openapi,
401: SearchTokenInvalidError().openapi, 401: SearchTokenInvalidError().openapi,
@ -547,7 +470,7 @@ async def photo_find(
current_user: User = Security(get_current_active_user, scopes=["photos.list"]), current_user: User = Security(get_current_active_user, scopes=["photos.list"]),
): ):
if token is not None: if token is not None:
found_record = await col_tokens.find_one({"token": token}) found_record = col_tokens.find_one({"token": token})
if found_record is None: if found_record is None:
raise SearchTokenInvalidError() raise SearchTokenInvalidError()
@ -564,7 +487,7 @@ async def photo_find(
current_user=current_user, current_user=current_user,
) )
if (await col_albums.find_one({"user": current_user.user, "name": album})) is None: if col_albums.find_one({"user": current_user.user, "name": album}) is None:
raise AlbumNameNotFoundError(album) raise AlbumNameNotFoundError(album)
if page <= 0 or page_size <= 0: if page <= 0 or page_size <= 0:
@ -593,7 +516,7 @@ async def photo_find(
} }
elif q is None and caption is None: elif q is None and caption is None:
raise PhotoSearchQueryEmptyError() raise PhotoSearchQueryEmptyError()
elif q is None: elif q is None and caption is not None:
db_query = { db_query = {
"user": current_user.user, "user": current_user.user,
"album": album, "album": album,
@ -604,7 +527,7 @@ async def photo_find(
"album": album, "album": album,
"caption": re.compile(caption), "caption": re.compile(caption),
} }
elif caption is None: elif q is not None and caption is None:
db_query = { db_query = {
"user": current_user.user, "user": current_user.user,
"album": album, "album": album,
@ -616,22 +539,16 @@ async def photo_find(
"filename": re.compile(q), "filename": re.compile(q),
} }
else: else:
db_query = { db_query = {"user": current_user.user, "album": album, "filename": re.compile(q), "caption": re.compile(caption)} # type: ignore
"user": current_user.user, db_query_count = {"user": current_user.user, "album": album, "filename": re.compile(q), "caption": re.compile(caption)} # type: ignore
"album": album,
"filename": re.compile(q),
"caption": re.compile(caption),
}
db_query_count = {
"user": current_user.user,
"album": album,
"filename": re.compile(q),
"caption": re.compile(caption),
}
async for image in col_photos.find(db_query, limit=page_size, skip=skip).sort( images = list(
"dates.uploaded", direction=DESCENDING col_photos.find(db_query, limit=page_size, skip=skip).sort(
): "dates.uploaded", DESCENDING
)
)
for image in images:
output["results"].append( output["results"].append(
{ {
"id": image["_id"].__str__(), "id": image["_id"].__str__(),
@ -640,9 +557,9 @@ async def photo_find(
} }
) )
if (await col_photos.count_documents(db_query_count)) > page * page_size: if col_photos.count_documents(db_query_count) > page * page_size:
token = str(token_urlsafe(32)) token = str(token_urlsafe(32))
await col_tokens.insert_one( col_tokens.insert_one(
{ {
"token": token, "token": token,
"query": q, "query": q,

12
extensions/security.py
View File

@ -1,10 +1,12 @@
from datetime import timedelta from datetime import timedelta
from fastapi import Depends
from fastapi.security import OAuth2PasswordRequestForm
from classes.exceptions import UserCredentialsInvalid from classes.exceptions import UserCredentialsInvalid
from modules.app import app from modules.app import app
from fastapi import Depends
from fastapi.security import (
OAuth2PasswordRequestForm,
)
from modules.security import ( from modules.security import (
ACCESS_TOKEN_EXPIRE_DAYS, ACCESS_TOKEN_EXPIRE_DAYS,
Token, Token,
@ -17,7 +19,7 @@ token_post_responses = {401: UserCredentialsInvalid().openapi}
@app.post("/token", response_model=Token, responses=token_post_responses) @app.post("/token", response_model=Token, responses=token_post_responses)
async def login_for_access_token(form_data: OAuth2PasswordRequestForm = Depends()): async def login_for_access_token(form_data: OAuth2PasswordRequestForm = Depends()):
user = await authenticate_user(form_data.username, form_data.password) user = authenticate_user(form_data.username, form_data.password)
if not user: if not user:
raise UserCredentialsInvalid() raise UserCredentialsInvalid()
access_token_expires = timedelta(days=ACCESS_TOKEN_EXPIRE_DAYS) access_token_expires = timedelta(days=ACCESS_TOKEN_EXPIRE_DAYS)

61
extensions/users.py
View File

@ -1,20 +1,27 @@
import logging
from datetime import datetime, timedelta from datetime import datetime, timedelta
from uuid import uuid1
from fastapi import Depends, Form
from fastapi.responses import Response, UJSONResponse
from starlette.status import HTTP_204_NO_CONTENT
from classes.exceptions import ( from classes.exceptions import (
UserAlreadyExists, UserAlreadyExists,
UserCredentialsInvalid, UserCredentialsInvalid,
UserEmailCodeInvalid, UserEmailCodeInvalid,
) )
from modules.database import (
col_users,
col_albums,
col_photos,
col_emails,
col_videos,
col_emails,
)
from modules.app import app from modules.app import app
from modules.database import col_albums, col_emails, col_photos, col_users, col_videos from modules.utils import configGet, logWrite
from modules.mailer import mail_sender
from modules.scheduler import scheduler from modules.scheduler import scheduler
from modules.mailer import mail_sender
from uuid import uuid1
from fastapi import Depends, Form
from fastapi.responses import Response, UJSONResponse
from starlette.status import HTTP_204_NO_CONTENT
from modules.security import ( from modules.security import (
User, User,
get_current_active_user, get_current_active_user,
@ -22,9 +29,6 @@ from modules.security import (
get_user, get_user,
verify_password, verify_password,
) )
from modules.utils import configGet
logger = logging.getLogger(__name__)
async def send_confirmation(user: str, email: str): async def send_confirmation(user: str, email: str):
@ -41,14 +45,12 @@ async def send_confirmation(user: str, email: str):
+ f"/users/{user}/confirm?code={confirmation_code}" + f"/users/{user}/confirm?code={confirmation_code}"
), ),
) )
await col_emails.insert_one( col_emails.insert_one(
{"user": user, "email": email, "used": False, "code": confirmation_code} {"user": user, "email": email, "used": False, "code": confirmation_code}
) )
logger.info( logWrite(f"Sent confirmation email to '{email}' with code {confirmation_code}")
"Sent confirmation email to '%s' with code %s", email, confirmation_code except Exception as exp:
) logWrite(f"Could not send confirmation email to '{email}' due to: {exp}")
except Exception as exc:
logger.error("Could not send confirmation email to '%s' due to: %s", email, exc)
@app.get("/users/me/", response_model=User) @app.get("/users/me/", response_model=User)
@ -80,15 +82,15 @@ if configGet("registration_requires_confirmation") is True:
responses=user_confirm_responses, responses=user_confirm_responses,
) )
async def user_confirm(user: str, code: str): async def user_confirm(user: str, code: str):
confirm_record = await col_emails.find_one( confirm_record = col_emails.find_one(
{"user": user, "code": code, "used": False} {"user": user, "code": code, "used": False}
) )
if confirm_record is None: if confirm_record is None:
raise UserEmailCodeInvalid() raise UserEmailCodeInvalid()
await col_emails.find_one_and_update( col_emails.find_one_and_update(
{"_id": confirm_record["_id"]}, {"$set": {"used": True}} {"_id": confirm_record["_id"]}, {"$set": {"used": True}}
) )
await col_users.find_one_and_update( col_users.find_one_and_update(
{"user": confirm_record["user"]}, {"$set": {"disabled": False}} {"user": confirm_record["user"]}, {"$set": {"disabled": False}}
) )
return UJSONResponse({"detail": configGet("email_confirmed", "messages")}) return UJSONResponse({"detail": configGet("email_confirmed", "messages")})
@ -103,13 +105,12 @@ if configGet("registration_enabled") is True:
async def user_create( async def user_create(
user: str = Form(), email: str = Form(), password: str = Form() user: str = Form(), email: str = Form(), password: str = Form()
): ):
if (await col_users.find_one({"user": user})) is not None: if col_users.find_one({"user": user}) is not None:
raise UserAlreadyExists() raise UserAlreadyExists()
await col_users.insert_one( col_users.insert_one(
{ {
"user": user, "user": user,
"email": email, "email": email,
"quota": None,
"hash": get_password_hash(password), "hash": get_password_hash(password),
"disabled": configGet("registration_requires_confirmation"), "disabled": configGet("registration_requires_confirmation"),
} }
@ -133,14 +134,14 @@ user_delete_responses = {401: UserCredentialsInvalid().openapi}
async def user_delete( async def user_delete(
password: str = Form(), current_user: User = Depends(get_current_active_user) password: str = Form(), current_user: User = Depends(get_current_active_user)
): ):
user = await get_user(current_user.user) user = get_user(current_user.user)
if not user: if not user:
return False return False
if not verify_password(password, user.hash): if not verify_password(password, user.hash):
raise UserCredentialsInvalid() raise UserCredentialsInvalid()
await col_users.delete_many({"user": current_user.user}) col_users.delete_many({"user": current_user.user})
await col_emails.delete_many({"user": current_user.user}) col_emails.delete_many({"user": current_user.user})
await col_photos.delete_many({"user": current_user.user}) col_photos.delete_many({"user": current_user.user})
await col_videos.delete_many({"user": current_user.user}) col_videos.delete_many({"user": current_user.user})
await col_albums.delete_many({"user": current_user.user}) col_albums.delete_many({"user": current_user.user})
return Response(status_code=HTTP_204_NO_CONTENT) return Response(status_code=HTTP_204_NO_CONTENT)

267
extensions/videos.py
View File

@ -1,44 +1,31 @@
import re import re
from datetime import datetime, timezone import pickle
from os import makedirs, remove
from pathlib import Path
from random import randint
from secrets import token_urlsafe from secrets import token_urlsafe
from shutil import move from shutil import move
from typing import Union from typing import Union
import aiofiles
from bson.errors import InvalidId
from bson.objectid import ObjectId
from fastapi import Security, UploadFile
from fastapi.responses import Response, UJSONResponse
from magic import Magic from magic import Magic
from pymongo import DESCENDING from datetime import datetime, timezone
from starlette.status import HTTP_204_NO_CONTENT from os import makedirs, path, remove
from classes.exceptions import ( from classes.exceptions import (
AlbumNameNotFoundError, AlbumNameNotFoundError,
SearchLimitInvalidError,
SearchPageInvalidError, SearchPageInvalidError,
SearchTokenInvalidError, SearchTokenInvalidError,
UserMediaQuotaReached,
VideoNotFoundError, VideoNotFoundError,
VideoSearchQueryEmptyError, VideoSearchQueryEmptyError,
) )
from classes.models import ( from classes.models import Video, SearchResultsVideo, VideoPublic
RandomSearchResultsVideo,
SearchResultsVideo,
Video,
VideoPublic,
)
from modules.app import app
from modules.database import col_albums, col_photos, col_tokens, col_videos
from modules.security import User, get_current_active_user from modules.security import User, get_current_active_user
from modules.app import app
from modules.database import col_videos, col_albums, col_tokens
from bson.objectid import ObjectId
from bson.errors import InvalidId
from pymongo import DESCENDING
video_post_responses = { from fastapi import UploadFile, Security
403: UserMediaQuotaReached().openapi, from fastapi.responses import UJSONResponse, Response
404: AlbumNameNotFoundError("name").openapi, from starlette.status import HTTP_204_NO_CONTENT
}
video_post_responses = {404: AlbumNameNotFoundError("name").openapi}
@app.post( @app.post(
@ -54,37 +41,34 @@ async def video_upload(
caption: Union[str, None] = None, caption: Union[str, None] = None,
current_user: User = Security(get_current_active_user, scopes=["videos.write"]), current_user: User = Security(get_current_active_user, scopes=["videos.write"]),
): ):
if (await col_albums.find_one({"user": current_user.user, "name": album})) is None: if col_albums.find_one({"user": current_user.user, "name": album}) is None:
raise AlbumNameNotFoundError(album) raise AlbumNameNotFoundError(album)
user_media_count = ( makedirs(
await col_videos.count_documents({"user": current_user.user}) path.join("data", "users", current_user.user, "albums", album), exist_ok=True
) + (await col_photos.count_documents({"user": current_user.user})) )
if user_media_count >= current_user.quota and not current_user.quota == -1: # type: ignore
raise UserMediaQuotaReached()
makedirs(Path(f"data/users/{current_user.user}/albums/{album}"), exist_ok=True)
filename = file.filename filename = file.filename
if Path(f"data/users/{current_user.user}/albums/{album}/{file.filename}").exists(): if path.exists(
path.join("data", "users", current_user.user, "albums", album, file.filename)
):
base_name = file.filename.split(".")[:-1] base_name = file.filename.split(".")[:-1]
extension = file.filename.split(".")[-1] extension = file.filename.split(".")[-1]
filename = ( filename = (
".".join(base_name) + f"_{int(datetime.now().timestamp())}." + extension ".".join(base_name) + f"_{int(datetime.now().timestamp())}." + extension
) )
async with aiofiles.open( with open(
Path(f"data/users/{current_user.user}/albums/{album}/{filename}"), "wb" path.join("data", "users", current_user.user, "albums", album, filename), "wb"
) as f: ) as f:
await f.write(await file.read()) f.write(await file.read())
# Hashing and duplicates check should be here # Hashing and duplicates check should be here
# Coords extraction should be here # Coords extraction should be here
uploaded = await col_videos.insert_one( uploaded = col_videos.insert_one(
{ {
"user": current_user.user, "user": current_user.user,
"album": album, "album": album,
@ -107,49 +91,31 @@ async def video_upload(
) )
video_get_responses = { video_get_responses = {404: VideoNotFoundError("id").openapi}
200: {
"content": {
"application/octet-stream": {
"schema": {
"type": "string",
"format": "binary",
"contentMediaType": "video/*",
}
}
}
},
404: VideoNotFoundError("id").openapi,
}
@app.get( @app.get("/videos/{id}", description="Get a video by id", responses=video_get_responses)
"/videos/{id}",
description="Get a video by id",
responses=video_get_responses,
response_class=Response,
)
async def video_get( async def video_get(
id: str, id: str,
current_user: User = Security(get_current_active_user, scopes=["videos.read"]), current_user: User = Security(get_current_active_user, scopes=["videos.read"]),
): ):
try: try:
video = await col_videos.find_one({"_id": ObjectId(id)}) video = col_videos.find_one({"_id": ObjectId(id)})
if video is None: if video is None:
raise InvalidId(id) raise InvalidId(id)
except InvalidId as exc: except InvalidId:
raise VideoNotFoundError(id) from exc raise VideoNotFoundError(id)
video_path = Path( video_path = path.join(
f"data/users/{current_user.user}/albums/{video['album']}/{video['filename']}" "data", "users", current_user.user, "albums", video["album"], video["filename"]
) )
mime = Magic(mime=True).from_file(video_path) mime = Magic(mime=True).from_file(video_path)
async with aiofiles.open(video_path, "rb") as f: with open(video_path, "rb") as f:
video_file = await f.read() video_file = f.read()
return Response(content=video_file, media_type=mime) return Response(video_file, media_type=mime)
video_move_responses = {404: VideoNotFoundError("id").openapi} video_move_responses = {404: VideoNotFoundError("id").openapi}
@ -167,18 +133,20 @@ async def video_move(
current_user: User = Security(get_current_active_user, scopes=["videos.write"]), current_user: User = Security(get_current_active_user, scopes=["videos.write"]),
): ):
try: try:
video = await col_videos.find_one({"_id": ObjectId(id)}) video = col_videos.find_one({"_id": ObjectId(id)})
if video is None: if video is None:
raise InvalidId(id) raise InvalidId(id)
except InvalidId as exc: except InvalidId:
raise VideoNotFoundError(id) from exc raise VideoNotFoundError(id)
if (await col_albums.find_one({"user": current_user.user, "name": album})) is None: if col_albums.find_one({"user": current_user.user, "name": album}) is None:
raise AlbumNameNotFoundError(album) raise AlbumNameNotFoundError(album)
if Path( if path.exists(
f"data/users/{current_user.user}/albums/{album}/{video['filename']}" path.join(
).exists(): "data", "users", current_user.user, "albums", album, video["filename"]
)
):
base_name = video["filename"].split(".")[:-1] base_name = video["filename"].split(".")[:-1]
extension = video["filename"].split(".")[-1] extension = video["filename"].split(".")[-1]
filename = ( filename = (
@ -187,7 +155,7 @@ async def video_move(
else: else:
filename = video["filename"] filename = video["filename"]
await col_videos.find_one_and_update( col_videos.find_one_and_update(
{"_id": ObjectId(id)}, {"_id": ObjectId(id)},
{ {
"$set": { "$set": {
@ -199,10 +167,15 @@ async def video_move(
) )
move( move(
Path( path.join(
f"data/users/{current_user.user}/albums/{video['album']}/{video['filename']}" "data",
"users",
current_user.user,
"albums",
video["album"],
video["filename"],
), ),
Path(f"data/users/{current_user.user}/albums/{album}/{filename}"), path.join("data", "users", current_user.user, "albums", album, filename),
) )
return UJSONResponse( return UJSONResponse(
@ -229,13 +202,13 @@ async def video_patch(
current_user: User = Security(get_current_active_user, scopes=["videos.write"]), current_user: User = Security(get_current_active_user, scopes=["videos.write"]),
): ):
try: try:
video = await col_videos.find_one({"_id": ObjectId(id)}) video = col_videos.find_one({"_id": ObjectId(id)})
if video is None: if video is None:
raise InvalidId(id) raise InvalidId(id)
except InvalidId as exc: except InvalidId:
raise VideoNotFoundError(id) from exc raise VideoNotFoundError(id)
await col_videos.find_one_and_update( col_videos.find_one_and_update(
{"_id": ObjectId(id)}, {"_id": ObjectId(id)},
{"$set": {"caption": caption, "dates.modified": datetime.now(tz=timezone.utc)}}, {"$set": {"caption": caption, "dates.modified": datetime.now(tz=timezone.utc)}},
) )
@ -263,84 +236,28 @@ async def video_delete(
current_user: User = Security(get_current_active_user, scopes=["videos.write"]), current_user: User = Security(get_current_active_user, scopes=["videos.write"]),
): ):
try: try:
video = await col_videos.find_one_and_delete({"_id": ObjectId(id)}) video = col_videos.find_one_and_delete({"_id": ObjectId(id)})
if video is None: if video is None:
raise InvalidId(id) raise InvalidId(id)
except InvalidId as exc: except InvalidId:
raise VideoNotFoundError(id) from exc raise VideoNotFoundError(id)
album = await col_albums.find_one({"name": video["album"]}) album = col_albums.find_one({"name": video["album"]})
remove( remove(
Path( path.join(
f"data/users/{current_user.user}/albums/{video['album']}/{video['filename']}" "data",
"users",
current_user.user,
"albums",
video["album"],
video["filename"],
) )
) )
return Response(status_code=HTTP_204_NO_CONTENT) return Response(status_code=HTTP_204_NO_CONTENT)
video_random_responses = {
400: SearchLimitInvalidError().openapi,
404: AlbumNameNotFoundError("name").openapi,
}
@app.get(
"/albums/{album}/videos/random",
description="Get one random video, optionally by caption",
response_class=UJSONResponse,
response_model=RandomSearchResultsVideo,
responses=video_random_responses,
)
async def video_random(
album: str,
caption: Union[str, None] = None,
limit: int = 100,
current_user: User = Security(get_current_active_user, scopes=["videos.list"]),
):
if (await col_albums.find_one({"user": current_user.user, "name": album})) is None:
raise AlbumNameNotFoundError(album)
if limit <= 0:
raise SearchLimitInvalidError()
output = {"results": []}
db_query = (
{
"user": current_user.user,
"album": album,
"caption": re.compile(caption),
}
if caption is not None
else {
"user": current_user.user,
"album": album,
}
)
documents_count = await col_videos.count_documents(db_query)
skip = randint(0, documents_count - 1) if documents_count > 1 else 0
async for video in col_videos.aggregate(
[
{"$match": db_query},
{"$skip": skip},
{"$limit": limit},
]
):
output["results"].append(
{
"id": video["_id"].__str__(),
"filename": video["filename"],
"caption": video["caption"],
}
)
return UJSONResponse(output)
video_find_responses = { video_find_responses = {
400: SearchPageInvalidError().openapi, 400: SearchPageInvalidError().openapi,
401: SearchTokenInvalidError().openapi, 401: SearchTokenInvalidError().openapi,
@ -366,7 +283,7 @@ async def video_find(
current_user: User = Security(get_current_active_user, scopes=["videos.list"]), current_user: User = Security(get_current_active_user, scopes=["videos.list"]),
): ):
if token is not None: if token is not None:
found_record = await col_tokens.find_one({"token": token}) found_record = col_tokens.find_one({"token": token})
if found_record is None: if found_record is None:
raise SearchTokenInvalidError() raise SearchTokenInvalidError()
@ -380,7 +297,7 @@ async def video_find(
current_user=current_user, current_user=current_user,
) )
if (await col_albums.find_one({"user": current_user.user, "name": album})) is None: if col_albums.find_one({"user": current_user.user, "name": album}) is None:
raise AlbumNameNotFoundError(album) raise AlbumNameNotFoundError(album)
if page <= 0 or page_size <= 0: if page <= 0 or page_size <= 0:
@ -392,7 +309,7 @@ async def video_find(
if q is None and caption is None: if q is None and caption is None:
raise VideoSearchQueryEmptyError() raise VideoSearchQueryEmptyError()
if q is None: if q is None and caption is not None:
db_query = { db_query = {
"user": current_user.user, "user": current_user.user,
"album": album, "album": album,
@ -403,34 +320,30 @@ async def video_find(
"album": album, "album": album,
"caption": re.compile(caption), "caption": re.compile(caption),
} }
elif caption is None: elif q is not None and caption is None:
db_query = { db_query = list(
"user": current_user.user, col_videos.find(
"album": album, {"user": current_user.user, "album": album, "filename": re.compile(q)},
"filename": re.compile(q), limit=page_size,
} skip=skip,
).sort("dates.uploaded", DESCENDING)
)
db_query_count = { db_query_count = {
"user": current_user.user, "user": current_user.user,
"album": album, "album": album,
"caption": re.compile(q), "caption": re.compile(q),
} }
else: else:
db_query = { db_query = list(col_videos.find({"user": current_user.user, "album": album, "filename": re.compile(q), "caption": re.compile(caption)}, limit=page_size, skip=skip).sort("dates.uploaded", DESCENDING)) # type: ignore
"user": current_user.user, db_query_count = {"user": current_user.user, "album": album, "filename": re.compile(q), "caption": re.compile(caption)} # type: ignore
"album": album,
"filename": re.compile(q),
"caption": re.compile(caption),
}
db_query_count = {
"user": current_user.user,
"album": album,
"filename": re.compile(q),
"caption": re.compile(caption),
}
async for video in col_videos.find(db_query, limit=page_size, skip=skip).sort( videos = list(
"dates.uploaded", direction=DESCENDING col_videos.find(db_query, limit=page_size, skip=skip).sort(
): "dates.uploaded", DESCENDING
)
)
for video in videos:
output["results"].append( output["results"].append(
{ {
"id": video["_id"].__str__(), "id": video["_id"].__str__(),
@ -439,9 +352,9 @@ async def video_find(
} }
) )
if (await col_videos.count_documents(db_query_count)) > page * page_size: if col_videos.count_documents(db_query_count) > page * page_size:
token = str(token_urlsafe(32)) token = str(token_urlsafe(32))
await col_tokens.insert_one( col_tokens.insert_one(
{ {
"token": token, "token": token,
"query": q, "query": q,

9
migrations/202311251700.py
View File

@ -1,9 +0,0 @@
from mongodb_migrations.base import BaseMigration
class Migration(BaseMigration):
def upgrade(self):
self.db.users.update_many({}, {"$set": {"quota": None}})
def downgrade(self):
self.db.test_collection.update_many({}, {"$unset": "quota"})

13
modules/app.py
View File

@ -1,14 +1,15 @@
from fastapi import FastAPI from fastapi import FastAPI
from fastapi.openapi.docs import get_redoc_html, get_swagger_ui_html from fastapi.openapi.docs import get_swagger_ui_html, get_redoc_html
app = FastAPI(title="END PLAY Photos", docs_url=None, redoc_url=None, version="0.6")
app = FastAPI(title="END PLAY Photos", docs_url=None, redoc_url=None, version="0.2")
@app.get("/docs", include_in_schema=False) @app.get("/docs", include_in_schema=False)
async def custom_swagger_ui_html(): async def custom_swagger_ui_html():
return get_swagger_ui_html( return get_swagger_ui_html(
openapi_url=app.openapi_url, openapi_url=app.openapi_url, # type: ignore
title=f"{app.title} - Documentation", title=app.title + " - Documentation",
swagger_favicon_url="/favicon.ico", swagger_favicon_url="/favicon.ico",
) )
@ -16,7 +17,7 @@ async def custom_swagger_ui_html():
@app.get("/redoc", include_in_schema=False) @app.get("/redoc", include_in_schema=False)
async def custom_redoc_html(): async def custom_redoc_html():
return get_redoc_html( return get_redoc_html(
openapi_url=app.openapi_url, openapi_url=app.openapi_url, # type: ignore
title=f"{app.title} - Documentation", title=app.title + " - Documentation",
redoc_favicon_url="/favicon.ico", redoc_favicon_url="/favicon.ico",
) )

15
modules/database.py
View File

@ -1,7 +1,5 @@
from async_pymongo import AsyncClient
from pymongo import GEOSPHERE, MongoClient
from modules.utils import configGet from modules.utils import configGet
from pymongo import MongoClient, GEOSPHERE
db_config = configGet("database") db_config = configGet("database")
@ -18,11 +16,16 @@ else:
db_config["host"], db_config["port"], db_config["name"] db_config["host"], db_config["port"], db_config["name"]
) )
db_client = AsyncClient(con_string) db_client = MongoClient(con_string)
db_client_sync = MongoClient(con_string)
db = db_client.get_database(name=db_config["name"]) db = db_client.get_database(name=db_config["name"])
collections = db.list_collection_names()
for collection in ["users", "albums", "photos", "videos", "tokens", "emails"]:
if not collection in collections:
db.create_collection(collection)
col_users = db.get_collection("users") col_users = db.get_collection("users")
col_albums = db.get_collection("albums") col_albums = db.get_collection("albums")
col_photos = db.get_collection("photos") col_photos = db.get_collection("photos")
@ -30,4 +33,4 @@ col_videos = db.get_collection("videos")
col_tokens = db.get_collection("tokens") col_tokens = db.get_collection("tokens")
col_emails = db.get_collection("emails") col_emails = db.get_collection("emails")
db_client_sync[db_config["name"]]["photos"].create_index([("location", GEOSPHERE)]) col_photos.create_index([("location", GEOSPHERE)])

14
modules/exif_reader.py
View File

@ -1,7 +1,3 @@
import contextlib
from pathlib import Path
from typing import Mapping, Union
from exif import Image from exif import Image
@ -16,14 +12,12 @@ def decimal_coords(coords: float, ref: str) -> float:
* float: Decimal degrees * float: Decimal degrees
""" """
decimal_degrees = coords[0] + coords[1] / 60 + coords[2] / 3600 decimal_degrees = coords[0] + coords[1] / 60 + coords[2] / 3600
if ref == "S" or ref == "W":
if ref in {"S", "W"}:
decimal_degrees = -decimal_degrees decimal_degrees = -decimal_degrees
return round(decimal_degrees, 5) return round(decimal_degrees, 5)
def extract_location(filepath: Union[str, Path]) -> Mapping[str, float]: def extract_location(filepath: str) -> dict:
"""Get location data from image """Get location data from image
### Args: ### Args:
@ -41,9 +35,11 @@ def extract_location(filepath: Union[str, Path]) -> Mapping[str, float]:
if img.has_exif is False: if img.has_exif is False:
return output return output
with contextlib.suppress(AttributeError): try:
output["lng"] = decimal_coords(img.gps_longitude, img.gps_longitude_ref) output["lng"] = decimal_coords(img.gps_longitude, img.gps_longitude_ref)
output["lat"] = decimal_coords(img.gps_latitude, img.gps_latitude_ref) output["lat"] = decimal_coords(img.gps_latitude, img.gps_latitude_ref)
output["alt"] = img.gps_altitude output["alt"] = img.gps_altitude
except AttributeError:
pass
return output return output

26
modules/extensions_loader.py
View File

@ -1,7 +1,5 @@
from importlib.util import module_from_spec, spec_from_file_location from importlib.util import module_from_spec, spec_from_file_location
from os import getcwd, path, walk from os import getcwd, path, walk
from pathlib import Path
from typing import Union
# ================================================================================= # =================================================================================
@ -12,21 +10,17 @@ def get_py_files(src):
cwd = getcwd() # Current Working directory cwd = getcwd() # Current Working directory
py_files = [] py_files = []
for root, dirs, files in walk(src): for root, dirs, files in walk(src):
py_files.extend( for file in files:
Path(f"{cwd}/{root}/{file}") for file in files if file.endswith(".py") if file.endswith(".py"):
) py_files.append(path.join(cwd, root, file))
return py_files return py_files
def dynamic_import(module_name: str, py_path: str): def dynamic_import(module_name, py_path):
try: try:
module_spec = spec_from_file_location(module_name, py_path) module_spec = spec_from_file_location(module_name, py_path)
if module_spec is None: module = module_from_spec(module_spec) # type: ignore
raise RuntimeError( module_spec.loader.exec_module(module) # type: ignore
f"Module spec from module name {module_name} and path {py_path} is None"
)
module = module_from_spec(module_spec)
module_spec.loader.exec_module(module)
return module return module
except SyntaxError: except SyntaxError:
print( print(
@ -34,15 +28,15 @@ def dynamic_import(module_name: str, py_path: str):
flush=True, flush=True,
) )
return return
except Exception as exc: except Exception as exp:
print(f"Could not load extension {module_name} due to {exc}", flush=True) print(f"Could not load extension {module_name} due to {exp}", flush=True)
return return
def dynamic_import_from_src(src: Union[str, Path], star_import=False): def dynamic_import_from_src(src, star_import=False):
my_py_files = get_py_files(src) my_py_files = get_py_files(src)
for py_file in my_py_files: for py_file in my_py_files:
module_name = Path(py_file).stem module_name = path.split(py_file)[-1][:-3]
print(f"Importing {module_name} extension...", flush=True) print(f"Importing {module_name} extension...", flush=True)
imported_module = dynamic_import(module_name, py_file) imported_module = dynamic_import(module_name, py_file)
if imported_module != None: if imported_module != None:

34
modules/hasher.py
View File

@ -1,15 +1,11 @@
from pathlib import Path from modules.database import col_photos
from typing import Any, List, Mapping, Union
import cv2
import numpy as np import numpy as np
from numpy.typing import NDArray from numpy.typing import NDArray
from scipy import spatial from scipy import spatial
import cv2
from modules.database import col_photos
def hash_array_to_hash_hex(hash_array) -> str: def hash_array_to_hash_hex(hash_array):
# convert hash array of 0 or 1 to hash string in hex # convert hash array of 0 or 1 to hash string in hex
hash_array = np.array(hash_array, dtype=np.uint8) hash_array = np.array(hash_array, dtype=np.uint8)
hash_str = "".join(str(i) for i in 1 * hash_array.flatten()) hash_str = "".join(str(i) for i in 1 * hash_array.flatten())
@ -20,18 +16,18 @@ def hash_hex_to_hash_array(hash_hex) -> NDArray:
# convert hash string in hex to hash values of 0 or 1 # convert hash string in hex to hash values of 0 or 1
hash_str = int(hash_hex, 16) hash_str = int(hash_hex, 16)
array_str = bin(hash_str)[2:] array_str = bin(hash_str)[2:]
return np.array(list(array_str), dtype=np.float32) return np.array([i for i in array_str], dtype=np.float32)
async def get_duplicates_cache(album: str) -> Mapping[str, Any]: def get_duplicates_cache(album: str) -> dict:
return { output = {}
photo["filename"]: [photo["_id"].__str__(), photo["hash"]] for photo in col_photos.find({"album": album}):
async for photo in col_photos.find({"album": album}) output[photo["filename"]] = [photo["_id"].__str__(), photo["hash"]]
} return output
async def get_phash(filepath: Union[str, Path]) -> str: async def get_phash(filepath: str) -> str:
img = cv2.imread(str(filepath)) img = cv2.imread(filepath)
# resize image and convert to gray scale # resize image and convert to gray scale
img = cv2.resize(img, (64, 64)) img = cv2.resize(img, (64, 64))
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
@ -52,14 +48,14 @@ async def get_phash(filepath: Union[str, Path]) -> str:
return hash_array_to_hash_hex(dct_block.flatten()) return hash_array_to_hash_hex(dct_block.flatten())
async def get_duplicates(hash_string: str, album: str) -> List[Mapping[str, Any]]: async def get_duplicates(hash: str, album: str) -> list:
duplicates = [] duplicates = []
cache = await get_duplicates_cache(album) cache = get_duplicates_cache(album)
for image_name, image_object in cache.items(): for image_name in cache.keys():
try: try:
distance = spatial.distance.hamming( distance = spatial.distance.hamming(
hash_hex_to_hash_array(cache[image_name][1]), hash_hex_to_hash_array(cache[image_name][1]),
hash_hex_to_hash_array(hash_string), hash_hex_to_hash_array(hash),
) )
except ValueError: except ValueError:
continue continue

24
modules/mailer.py
View File

@ -1,11 +1,7 @@
import logging
from smtplib import SMTP, SMTP_SSL from smtplib import SMTP, SMTP_SSL
from ssl import create_default_context
from traceback import print_exc from traceback import print_exc
from ssl import create_default_context
from modules.utils import configGet from modules.utils import configGet, logWrite
logger = logging.getLogger(__name__)
try: try:
if configGet("use_ssl", "mailer", "smtp") is True: if configGet("use_ssl", "mailer", "smtp") is True:
@ -13,7 +9,7 @@ try:
configGet("host", "mailer", "smtp"), configGet("host", "mailer", "smtp"),
configGet("port", "mailer", "smtp"), configGet("port", "mailer", "smtp"),
) )
logger.info("Initialized SMTP SSL connection") logWrite(f"Initialized SMTP SSL connection")
elif configGet("use_tls", "mailer", "smtp") is True: elif configGet("use_tls", "mailer", "smtp") is True:
mail_sender = SMTP( mail_sender = SMTP(
configGet("host", "mailer", "smtp"), configGet("host", "mailer", "smtp"),
@ -21,21 +17,21 @@ try:
) )
mail_sender.starttls(context=create_default_context()) mail_sender.starttls(context=create_default_context())
mail_sender.ehlo() mail_sender.ehlo()
logger.info("Initialized SMTP TLS connection") logWrite(f"Initialized SMTP TLS connection")
else: else:
mail_sender = SMTP( mail_sender = SMTP(
configGet("host", "mailer", "smtp"), configGet("port", "mailer", "smtp") configGet("host", "mailer", "smtp"), configGet("port", "mailer", "smtp")
) )
mail_sender.ehlo() mail_sender.ehlo()
logger.info("Initialized SMTP connection") logWrite(f"Initialized SMTP connection")
except Exception as exc: except Exception as exp:
logger.error("Could not initialize SMTP connection to: %s", exc) logWrite(f"Could not initialize SMTP connection to: {exp}")
print_exc() print_exc()
try: try:
mail_sender.login( mail_sender.login(
configGet("login", "mailer", "smtp"), configGet("password", "mailer", "smtp") configGet("login", "mailer", "smtp"), configGet("password", "mailer", "smtp")
) )
logger.info("Successfully initialized mailer") logWrite(f"Successfully initialized mailer")
except Exception as exc: except Exception as exp:
logger.error("Could not login into provided SMTP account due to: %s", exc) logWrite(f"Could not login into provided SMTP account due to: {exp}")

23
modules/migrator.py
View File

@ -1,23 +0,0 @@
from typing import Any, Mapping
from mongodb_migrations.cli import MigrationManager
from mongodb_migrations.config import Configuration
from modules.utils import configGet
def migrate_database() -> None:
"""Apply migrations from folder `migrations/` to the database"""
db_config: Mapping[str, Any] = configGet("database")
manager_config = Configuration(
{
"mongo_host": db_config["host"],
"mongo_port": db_config["port"],
"mongo_database": db_config["name"],
"mongo_username": db_config["user"],
"mongo_password": db_config["password"],
}
)
manager = MigrationManager(manager_config)
manager.run()

83
modules/security.py
View File

@ -1,34 +1,19 @@
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from os import getenv
from typing import List, Union from typing import List, Union
from modules.database import col_users
from fastapi import Depends, HTTPException, Security, status from fastapi import Depends, HTTPException, Security, status
from fastapi.security import OAuth2PasswordBearer, SecurityScopes from fastapi.security import (
OAuth2PasswordBearer,
SecurityScopes,
)
from jose import JWTError, jwt from jose import JWTError, jwt
from passlib.context import CryptContext from passlib.context import CryptContext
from pydantic import BaseModel, ValidationError from pydantic import BaseModel, ValidationError
from modules.database import col_users
from modules.utils import configGet
try:
configGet("secret")
except KeyError as exc:
raise KeyError(
"PhotosAPI secret is not set. Secret key handling has changed in PhotosAPI 0.6.0, so you need to add the config key 'secret' to your config file."
) from exc
if configGet("secret") == "" and getenv("PHOTOSAPI_SECRET") is None:
raise KeyError(
"PhotosAPI secret is not set. Set the config key 'secret' or provide the environment variable 'PHOTOSAPI_SECRET' containing a secret string."
)
SECRET_KEY = (
getenv("PHOTOSAPI_SECRET")
if getenv("PHOTOSAPI_SECRET") is not None
else configGet("secret")
)
with open("secret_key", "r", encoding="utf-8") as f:
SECRET_KEY = f.read()
ALGORITHM = "HS256" ALGORITHM = "HS256"
ACCESS_TOKEN_EXPIRE_DAYS = 180 ACCESS_TOKEN_EXPIRE_DAYS = 180
@ -46,7 +31,6 @@ class TokenData(BaseModel):
class User(BaseModel): class User(BaseModel):
user: str user: str
email: Union[str, None] = None email: Union[str, None] = None
quota: Union[int, None] = None
disabled: Union[bool, None] = None disabled: Union[bool, None] = None
@ -73,58 +57,49 @@ oauth2_scheme = OAuth2PasswordBearer(
) )
def verify_password(plain_password, hashed_password) -> bool: def verify_password(plain_password, hashed_password):
return pwd_context.verify(plain_password, hashed_password) return pwd_context.verify(plain_password, hashed_password)
def get_password_hash(password) -> str: def get_password_hash(password):
return pwd_context.hash(password) return pwd_context.hash(password)
async def get_user(user: str) -> UserInDB: def get_user(user: str):
found_user = await col_users.find_one({"user": user}) found_user = col_users.find_one({"user": user})
if found_user is None:
raise RuntimeError(f"User {user} does not exist")
return UserInDB( return UserInDB(
user=found_user["user"], user=found_user["user"],
email=found_user["email"], email=found_user["email"],
quota=found_user["quota"]
if found_user["quota"] is not None
else configGet("default_user_quota"),
disabled=found_user["disabled"], disabled=found_user["disabled"],
hash=found_user["hash"], hash=found_user["hash"],
) )
async def authenticate_user(user_name: str, password: str) -> Union[UserInDB, bool]: def authenticate_user(user_name: str, password: str):
if user := await get_user(user_name): user = get_user(user_name)
return user if verify_password(password, user.hash) else False if not user:
else:
return False return False
if not verify_password(password, user.hash):
return False
return user
def create_access_token( def create_access_token(data: dict, expires_delta: Union[timedelta, None] = None):
data: dict, expires_delta: Union[timedelta, None] = None
) -> str:
to_encode = data.copy() to_encode = data.copy()
if expires_delta: if expires_delta:
expire = datetime.now(tz=timezone.utc) + expires_delta expire = datetime.now(tz=timezone.utc) + expires_delta
else: else:
expire = datetime.now(tz=timezone.utc) + timedelta( expire = datetime.now(tz=timezone.utc) + timedelta(
days=ACCESS_TOKEN_EXPIRE_DAYS days=ACCESS_TOKEN_EXPIRE_DAYS
) )
to_encode.update({"exp": expire})
to_encode["exp"] = expire encoded_jwt = jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM)
return encoded_jwt
return jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM)
async def get_current_user( async def get_current_user(
security_scopes: SecurityScopes, token: str = Depends(oauth2_scheme) security_scopes: SecurityScopes, token: str = Depends(oauth2_scheme)
) -> UserInDB: ):
if security_scopes.scopes: if security_scopes.scopes:
authenticate_value = f'Bearer scope="{security_scopes.scope_str}"' authenticate_value = f'Bearer scope="{security_scopes.scope_str}"'
else: else:
@ -139,18 +114,16 @@ async def get_current_user(
try: try:
payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM]) payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])
user: str = payload.get("sub") user: str = payload.get("sub")
if user is None: if user is None:
raise credentials_exception raise credentials_exception
token_scopes = payload.get("scopes", []) token_scopes = payload.get("scopes", [])
token_data = TokenData(scopes=token_scopes, user=user) token_data = TokenData(scopes=token_scopes, user=user)
except (JWTError, ValidationError) as exc: except (JWTError, ValidationError):
raise credentials_exception from exc raise credentials_exception
user_record = await get_user(user=token_data.user) user = get_user(user=token_data.user)
if user_record is None: if user is None:
raise credentials_exception raise credentials_exception
for scope in security_scopes.scopes: for scope in security_scopes.scopes:
@ -160,8 +133,7 @@ async def get_current_user(
detail="Not enough permissions", detail="Not enough permissions",
headers={"WWW-Authenticate": authenticate_value}, headers={"WWW-Authenticate": authenticate_value},
) )
return user
return user_record
async def get_current_active_user( async def get_current_active_user(
@ -169,5 +141,4 @@ async def get_current_active_user(
): ):
if current_user.disabled: if current_user.disabled:
raise HTTPException(status_code=400, detail="Inactive user") raise HTTPException(status_code=400, detail="Inactive user")
return current_user return current_user

42
modules/utils.py
View File

@ -1,18 +1,20 @@
import logging
from pathlib import Path
from traceback import format_exc
from typing import Any, Union from typing import Any, Union
from ujson import loads, dumps, JSONDecodeError
from ujson import JSONDecodeError, dumps, loads from traceback import print_exc
logger = logging.getLogger(__name__)
def jsonLoad(filepath: Union[str, Path]) -> Any: # Print to stdout and then to log
def logWrite(message: str, debug: bool = False) -> None:
# save to log file and rotation is to be done
# logAppend(f'{message}', debug=debug)
print(f"{message}", flush=True)
def jsonLoad(filepath: str) -> Any:
"""Load json file """Load json file
### Args: ### Args:
* filepath (`Union[str, Path]`): Path to input file * filepath (`str`): Path to input file
### Returns: ### Returns:
* `Any`: Some json deserializable * `Any`: Some json deserializable
@ -21,36 +23,32 @@ def jsonLoad(filepath: Union[str, Path]) -> Any:
try: try:
output = loads(file.read()) output = loads(file.read())
except JSONDecodeError: except JSONDecodeError:
logger.error( logWrite(
"Could not load json file %s: file seems to be incorrect!\n%s", f"Could not load json file {filepath}: file seems to be incorrect!\n{print_exc()}"
filepath,
format_exc(),
) )
raise raise
except FileNotFoundError: except FileNotFoundError:
logger.error( logWrite(
"Could not load json file %s: file does not seem to exist!\n%s", f"Could not load json file {filepath}: file does not seem to exist!\n{print_exc()}"
filepath,
format_exc(),
) )
raise raise
file.close() file.close()
return output return output
def jsonSave(contents: Union[list, dict], filepath: Union[str, Path]) -> None: def jsonSave(contents: Union[list, dict], filepath: str) -> None:
"""Save contents into json file """Save contents into json file
### Args: ### Args:
* contents (`Union[list, dict]`): Some json serializable * contents (`Union[list, dict]`): Some json serializable
* filepath (`Union[str, Path]`): Path to output file * filepath (`str`): Path to output file
""" """
try: try:
with open(filepath, "w", encoding="utf8") as file: with open(filepath, "w", encoding="utf8") as file:
file.write(dumps(contents, ensure_ascii=False, indent=4)) file.write(dumps(contents, ensure_ascii=False, indent=4))
file.close() file.close()
except Exception as exc: except Exception as exp:
logger.error("Could not save json file %s: %s\n%s", filepath, exc, format_exc()) logWrite(f"Could not save json file {filepath}: {exp}\n{print_exc()}")
return return
@ -64,7 +62,7 @@ def configGet(key: str, *args: str) -> Any:
### Returns: ### Returns:
* `Any`: Value of provided key * `Any`: Value of provided key
""" """
this_dict = jsonLoad(Path("config.json")) this_dict = jsonLoad("config.json")
this_key = this_dict this_key = this_dict
for dict_key in args: for dict_key in args:
this_key = this_key[dict_key] this_key = this_key[dict_key]

35
photos_api.py
View File

@ -1,22 +1,11 @@
import logging from os import makedirs, path
from argparse import ArgumentParser from modules.app import app
from os import makedirs from modules.utils import *
from pathlib import Path from modules.scheduler import scheduler
from modules.extensions_loader import dynamic_import_from_src
from fastapi.responses import FileResponse from fastapi.responses import FileResponse
from modules.app import app makedirs(path.join("data", "users"), exist_ok=True)
from modules.extensions_loader import dynamic_import_from_src
from modules.migrator import migrate_database
from modules.scheduler import scheduler
makedirs(Path("data/users"), exist_ok=True)
logging.basicConfig(
level=logging.INFO,
format="%(name)s.%(funcName)s | %(levelname)s | %(message)s",
datefmt="[%X]",
)
@app.get("/favicon.ico", response_class=FileResponse, include_in_schema=False) @app.get("/favicon.ico", response_class=FileResponse, include_in_schema=False)
@ -29,15 +18,3 @@ dynamic_import_from_src("extensions", star_import=True)
# ================================================================================= # =================================================================================
scheduler.start() scheduler.start()
parser = ArgumentParser(
prog="PhotosAPI",
description="Small and simple API server for saving photos and videos.",
)
parser.add_argument("--migrate", action="store_true")
args, unknown = parser.parse_known_args()
if args.migrate:
migrate_database()

20
requirements.txt
View File

@ -1,14 +1,10 @@
aiofiles==23.2.1 fastapi[all]==0.95.0
pymongo==4.3.3
ujson~=5.7.0
scipy~=1.10.1
python-magic~=0.4.27
opencv-python~=4.7.0.72
python-jose[cryptography]~=3.3.0
passlib~=1.7.4
apscheduler~=3.10.1 apscheduler~=3.10.1
exif==1.6.0 exif==1.6.0
fastapi[all]==0.104.1
mongodb-migrations==1.3.0
opencv-python~=4.8.1.78
passlib~=1.7.4
pymongo>=4.3.3
python-jose[cryptography]~=3.3.0
python-magic~=0.4.27
scipy~=1.11.0
ujson~=5.8.0
--extra-index-url https://git.end-play.xyz/api/packages/profitroll/pypi/simple
async_pymongo==0.1.4