proboards_scraper.database¶

class proboards_scraper.database.Database(db_path)[source]¶

This class serves as an interface for the SQLite database, and allows items to be inserted/updated or queried using a variety of specific functions that abstract away implementation details of the database and its schema.

Parameters: db_path (pathlib.Path) – Path to SQLite database file.

insert(obj, filters=None, update=False)[source]¶

Query the database for an object of the given sqlalchemy Metaclass using the given filters to determine if it already exists in the database. If it doesn’t, insert it into the database. Either way, return a bool indicating whether the object was added, as well as the resulting object from the query.

Although this method can be called directly, it is preferable to call the corresponding insert_* or query_* wrapper methods instead, which simplify the task of querying/inserting into the database.

Parameters

obj (sqlalchemy.orm.decl_api.DeclarativeMeta) –
A sqlalchemy Metaclass instance corresponding to a database table class, i.e., an instance of one of:
- Avatar
- Board
- Category
- CSS
- Image
- Moderator
- Poll
- PollOption
- PollVoter
- Post
- ShoutboxPost
- Thread
- User
filters (Optional[dict]) – A dict of key/value pairs on which to filter the query results. The keys should correspond to the attributes of the Metaclass, i.e., attributes of the obj argument class. See example below. If filters is None, it defaults to the id attribute of obj, i.e., obj.id.
update (bool) – Whether to update the database entry if the queried object already exists.

Example

The following example demonstrates how to insert a new user into the database. Note that we first create an instance of the user (which is passed to insert()) and filter by the user id (which is the User table primary key). In other words, this searches the database for an existing user with the given filter (i.e., the user with id 7) and, if the user doesn’t exist, inserts it into the database, then returns the inserted object:

user_data = {
    "id": 7,
    "date_registered": 1631019126,
    "email": "foo@bar.com",
    "name": "Snake Plissken",
    "username": "snake",
}

new_user = User(**user_data)

db = Database("forum.db")
inserted, new_user = db.insert(
    new_user,
    filters={"id": 7}
)

Return type

Tuple[int, sqlalchemy.orm.decl_api.DeclarativeMeta]

Returns

(inserted, ret)

inserted:
An integer code denoting insert status.
- 0: The object failed to be inserted or updated.
- 1: obj was inserted into the database.
- 2: obj existed in the database and was updated.
ret:
The inserted object, if the object didn’t previously exist in the database, or the existing object if it did already exist. It is effectively an updated version of obj.

insert_avatar(avatar_, update=False)[source]¶

Insert a user avatar into the database; this method wraps insert().

Parameters

avatar_ (dict) – A dict containing the keyword args (attributes) needed to instantiate a Avatar object.
update (bool) – See insert().

Return type

proboards_scraper.database.Avatar

Returns

The inserted (or updated) Avatar object.

insert_board(board_, update=False)[source]¶

Insert a board into the database; this method wraps insert().

Parameters

board_ (dict) – A dict containing the keyword args (attributes) needed to instantiate a Board object.
update (bool) – See insert().

Return type

proboards_scraper.database.Board

Returns

The inserted (or updated) Board object.

insert_category(category_, update=False)[source]¶

Insert a category into the database; this method wraps insert().

Parameters

category_ (dict) – A dict containing the keyword args (attributes) needed to instantiate a Category object.
update (bool) – See insert().

Return type

proboards_scraper.database.Category

Returns

The inserted (or updated) Category object.

insert_guest(guest_)[source]¶

Guest users are a special case of User. Guests are users who do not have a user id or a user profile page, and may include deleted users. Since there may be posts or threads started by guests, they are treated as normal users for the purposes of the database, except they are assigned a negative integer user id (which does not exist on the actual forum). Because a given guest has only a username and not a user id, guests are queried by name. If a guest does not already exist in the database, the next smallest negative integer is used as their user id.

Parameters: guest_ (dict) – A dict containing a name key, corresponding to the guest user’s name.
Return type: proboards_scraper.database.User
Returns: The inserted or existing User object corresponding to the guest.

insert_image(image_, update=False)[source]¶

Insert an image into the database; this method wraps insert().

Parameters

image_ (dict) – A dict containing the keyword args (attributes) needed to instantiate a Image object.
update (bool) – See insert().

Return type

proboards_scraper.database.Image

Returns

The inserted (or updated) Image object.

insert_moderator(moderator_, update=False)[source]¶

Insert a moderator into the database; this method wraps insert().

Parameters

moderator_ (dict) – A dict containing the keyword args (attributes) needed to instantiate a Moderator object.
update (bool) – See insert().

Return type

proboards_scraper.database.Moderator

Returns

The inserted (or updated) Moderator object.

insert_poll(poll_, update=False)[source]¶

Insert a poll into the database; this method wraps insert().

Parameters

poll_ (dict) – A dict containing the keyword args (attributes) needed to instantiate a Poll object.
update (bool) – See insert().

Return type

proboards_scraper.database.Poll

Returns

The inserted (or updated) Poll object.

insert_poll_option(poll_option_, update=False)[source]¶

Insert a poll option into the database; this method wraps insert().

Parameters

poll_option_ (dict) – A dict containing the keyword args (attributes) needed to instantiate a PollOption object.
update (bool) – See insert().

Return type

proboards_scraper.database.PollOption

Returns

The inserted (or updated) PollOption object.

insert_poll_voter(poll_voter_, update=False)[source]¶

Insert a poll voter into the database; this method wraps insert().

Parameters

poll_voter_ (dict) – A dict containing the keyword args (attributes) needed to instantiate a PollVoter object.
update (bool) – See insert().

Return type

proboards_scraper.database.PollVoter

Returns

The inserted (or updated) PollVoter object.

insert_post(post_, update=False)[source]¶

Insert a post into the database; this method wraps insert().

Parameters

post_ (dict) – A dict containing the keyword args (attributes) needed to instantiate a Post object.
update (bool) – See insert().

Return type

proboards_scraper.database.Post

Returns

The inserted (or updated) Post object.

insert_shoutbox_post(shoutbox_post_, update=False)[source]¶

Insert a shoutbox post into the database; this method wraps insert().

Parameters

shoutbox_post_ (dict) – A dict containing the keyword args (attributes) needed to instantiate a ShoutboxPost object.
update (bool) – See insert().

Return type

proboards_scraper.database.ShoutboxPost

Returns

The inserted (or updated) ShoutboxPost object.

insert_thread(thread_, update=False)[source]¶

Insert a thread into the database; this method wraps insert().

Parameters

thread_ (dict) – A dict containing the keyword args (attributes) needed to instantiate a Thread object.
update (bool) – See insert().

Return type

proboards_scraper.database.Thread

Returns

The inserted (or updated) Thread object.

insert_user(user_, update=False)[source]¶

Insert a user into the database; this method wraps insert().

Parameters

user_ (dict) – A dict containing the keyword args (attributes) needed to instantiate a User object.
update (bool) – See insert().

Return type

proboards_scraper.database.User

Returns

The inserted (or updated) User object.

query_boards(board_id=None)[source]¶

Return a list of all boards if no board_id is provided or a specific board if it is provided.

Parameters: board_id (Optional[int]) – A board id (optional).
Return type: Union[List[dict], dict]
Returns: A dict corresponding to a board in the database (if board_id was provided), else a list of dicts of all boards (if board_id was not provided).

Note

The returned Board object(s) are serialized to a human-readable JSON format (Python dict) by serialize().

query_threads(thread_id=None)[source]¶

Return a list of all threads if no thread_id is provided or a specific thread if it is provided.

Parameters: thread_id (Optional[int]) – A thread id (optional).
Return type: Union[List[dict], dict]
Returns: A dict corresponding to a thread in the database (if thread_id was provided), else a list of dicts of all threads (if thread_id was not provided).

Note

The returned Thread object(s) are serialized to a human-readable JSON format (Python dict) by serialize().

query_users(user_id=None)[source]¶

Return a list of all users, if no user_id is provided, or a specific user, if it is provided.

Parameters: user_id (Optional[int]) – A user id (optional).
Return type: Union[List[dict], dict]
Returns: A dict corresponding to a user in the database (if user_id was provided), else a list of dicts of all users (if user_id was not provided).

Note

The returned User object(s) are serialized to a human-readable JSON format (Python dict) by serialize().

class proboards_scraper.database.Avatar(**kwargs)[source]¶

Bases: sqlalchemy.orm.decl_api.Base

This table links a user to their avatar.

image_id¶

Image id of the image that corresponds to this avatar; see Image.

Type: int

user_id¶

User id of the user to which this avatar belongs; see User.

Type: int

class proboards_scraper.database.Board(**kwargs)[source]¶

Bases: sqlalchemy.orm.decl_api.Base

This table contains information on boards and their associated metadata.

id¶

Board number obtained from the board URL, eg, https://yoursite.proboards.com/board/42/general refers to the “General” board with id 42.

Type: int

category_id¶

Category id of the category to which this board belongs; see Category.

Type: int

description¶

Board description.

Type: str

name¶

Board name. Required.

Type: str

parent_id¶

Board id of this board’s parent board, if it is a sub-board.

Type: int

password_protected¶

Whether the board is password-protected.

Type: bool

url¶

Board URL.

Type: str

moderators¶: List of this board’s moderators, if any; see Moderator.

sub_boards¶: List of this board’s sub-boards, if any.

threads¶: List of this board’s threads; see Thread.

class proboards_scraper.database.Category(**kwargs)[source]¶

Bases: sqlalchemy.orm.decl_api.Base

This table stores information on categories (on the main page) and their associated metadata.

id¶

Category id number obtained from the main page source.

Type: int

name¶

Category name.

Type: str

boards¶: List of boards belonging to this category; see Board.

class proboards_scraper.database.CSS(**kwargs)[source]¶

Bases: sqlalchemy.orm.decl_api.Base

Table for storing information related to downloaded CSS files. The CSS files themselves should be stored on disk.

id¶

An arbitrary autoincrementing primary key for each CSS file.

Type: int

description¶

Description of the file.

Type: str

filename¶

Filename of the CSS file stored on disk.

Type: str

md5_hash¶

MD5 hash of the downloaded file.

Type: str

url¶

Original URL of the CSS file.

Type: str

class proboards_scraper.database.Image(**kwargs)[source]¶

Bases: sqlalchemy.orm.decl_api.Base

This table stores generic metadata for any image. Image files themselves should be downloaded and stored somewhere; this table only records the filename of the downloaded file (which may differ from the original filename, found in the url). The table may also be used to store metadata on files that no longer exist, e.g., an avatar hosted on a site that no longer exists, as a record of the original URL.

id¶

An arbitrary autoincrementing primary key for each image.

Type: int

description¶

Description of the image. Optional.

Type: str

filename¶

Filename of the downloaded file on disk.

Type: str

md5_hash¶

MD5 hash of the downloaded file.

Type: str

size¶

Size, in bytes, of the downloaded file.

Type: int

url¶

Original URL of the file.

Type: str