GitHub Common Lisp Weekly Trending

open-goal/jak-project

Reviving the language that brought us the Jak & Daxter Series

Please read first

Important

Our repositories on GitHub are for development of the project and tracking active issues. Most of the information you will find here pertains to setting up the project for development purposes and is not relevant to the end-user.

For a setup guide on how to install and play the game there is the following video that you can check out: https://youtu.be/K84UUMnkJc4

For questions or additional information pertaining to the project, we have a Discord for discussion here: https://discord.gg/VZbXMHXzWv

Additionally, you can find further documentation and answers to frequently asked questions on the project's main website: https://opengoal.dev

Warning

Do not use this decompilation project without the use of your own legally purchased copy of the game. OpenGOAL does not include any assets from the original games, so you must provide your own legitimately obtained PS2 copy of the game. OpenGOAL supports every retail PAL, NTSC, and NTSC-J build, including Greatest Hits copies. Please note that does NOT include any of the later releases (PS3/PS4/PS5).

Project Description

The project's goal is to port the original trilogy (Jak 1 -> Jak 3) to PC. Over 98% of the games were written in GOAL, a custom LISP language developed by Naughty Dog. Our strategy is:

decompile the original game code into human-readable GOAL code
develop our own compiler for GOAL and recompile the game code for x86-64
create a tool to extract game assets into formats that can be easily viewed or modified
create tools to repack game assets into a format that our port uses.

Our objectives are:

make the port a "native application" on x86-64, with high performance. It shouldn't be emulated, interpreted, or transpiled.
Our GOAL compiler's performance should be around the same as unoptimized C.
try to match things from the original game and development as possible. For example, the original GOAL compiler supported live modification of code while the game is running, so we do the same, even though it's not required for just porting the game.
support modifications. It should be possible to make edits to the code without everything else breaking.

At the moment we support x86_64 on Windows, Linux and macOS (via Rosetta translation). There are no plans to ever make a mobile release.

Current Status

Jak 1 has been considered in a polished, complete state for years at this point.
Jak 2 is considered in beta due to a few issues we are aware of that need fixing, however to the casual user, the game is essentially complete.
Jak 3 has a good amount of work left to do.

YouTube playlist showcasing some of the early progress for Jak 1: https://www.youtube.com/playlist?list=PLWx9T30aAT50cLnCTY1SAbt2TtWQzKfXX

Methodology

To assist with decompiling, we've built a decompiler that can process GOAL code and unpack game assets. We manually specify function types and locations where we believe the original code had type casts (or where they feel appropriate) until the decompilation succeeds, then we clean up the output of the decompiled code by adding comments and adjusting formatting, then save it in goal_src/.

Our decompiler is designed specifically for processing the output of the original GOAL compiler. As a result, when given correct casts, it often produces code that can be directly fed into a compiler and works perfectly. This is continually tested as part of our unit tests.

Setting up a Development Environment

The remainder of this README is aimed at people interested in building the project from source, typically with the intention of contributing as a developer.

If this does not sound like you and you just want to play the game, refer to the above section Quick Start

OS Setup

Editor Setup

You can of course use whatever editor you want, but here is some documentation that should help you get started on some of the editor's we have used and have written about:

Building and Running the Game

Getting a running game involves 4 main steps:

Build C++ tools (follow Getting Started steps above for your platform)
Extract assets from the game
Build the game
Run the game

Extract Assets

First, we have to setup our environment so we know which game and version we are operating with. For the black label version of Jak 1 we would run the following:

task set-game-jak1
task set-decomp-ntscv1 # or for example for PAL, `task set-decomp-pal`

Run task --list to see the other available options

Next, ensure you extract your ISO file contents into the relevant iso_data/<game-name> folder. In the case of Jak 1 this is iso_data/jak1.

Once this is done, open a terminal in the jak-project folder and run the following:

task extract

Build the Game (Running the Compiler)

The next step is to build the game itself. To do so, in the same terminal run the following:

task repl

You will be greeted with a prompt like so:

 _____             _____ _____ _____ __
|     |___ ___ ___|   __|     |  _  |  |
|  |  | . | -_|   |  |  |  |  |     |  |__
|_____|  _|___|_|_|_____|_____|__|__|_____|
      |_|
Welcome to OpenGOAL 0.8!
Run (repl-help) for help with common commands and REPL usage.
Run (lt) to connect to the local target.

g >

Run the following to build the game:

g > (mi)

IMPORTANT NOTE! If you're not using the non-default version of the game, you may hit issues trying to run (mi) in this step. An example error might include something like:

Input file iso_data/jak1/MUS/TWEAKVAL.MUS does not exist.

This is because the decompiler inputs/outputs using the gameName JSON field in the decompiler config. For example if you are using Jak 1 PAL, it will assume iso_data/jak1_pal and decompiler_out/jak1_pal. Therefore, you can inform the REPL/compiler of this via the gameVersionFolder config field described here

Run the Game

Finally the game can be launched. Open a second terminal from the jak-project directory and run the following:

task boot-game

The game should boot automatically if everything was done correctly.

Connecting the REPL to the Game

Connecting the REPL to the game allows you to inspect and modify code or data while the game is running.

To do so, in the REPL after a successful (mi), run the following:

g > (lt)

If successful, your prompt should change to:

gc>

For example, running the following will print out some basic information about Jak:

gc> *target*

Running the Game Without Auto-Booting

You can also start up the game without booting. To do so run the following in one terminal

task run-game

And then in your REPL run the following (after a successful (mi)):

g > (lt)
[Listener] Socket connected established! (took 0 tries). Waiting for version...
Got version 0.8 OK!
[Debugger] Context: valid = true, s7 = 0x147d24, base = 0x2123000000, tid = 2438049

gc> (lg)
10836466        #xa559f2              0.0000        ("game" "kernel")

gc> (test-play)
(play :use-vis #t :init-game #f) has been called!
0        #x0              0.0000        0

gc>

Technical Project Overview

Some more detail about the various components of the project can be found here

dimitri/pgloader

Migrate to PostgreSQL in a single command!

PGLoader

pgloader is a data loading tool for PostgreSQL, using the COPY command.

Its main advantage over just using COPY or \copy, and over using a Foreign Data Wrapper, is its transaction behaviour, where pgloader will keep a separate file of rejected data, but continue trying to copy good data in your database.

The default PostgreSQL behaviour is transactional, which means that any erroneous line in the input data (file or remote database) will stop the entire bulk load for the table.

pgloader also implements data reformatting, a typical example of that being the transformation of MySQL datestamps 0000-00-00 and 0000-00-00 00:00:00 to PostgreSQL NULL value (because our calendar never had a year zero).

Documentation

Full documentation is available online, including manual pages of all the pgloader sub-commands. Check out https://pgloader.readthedocs.io/.

$ pgloader --help
pgloader [ option ... ] SOURCE TARGET
  --help -h                       boolean  Show usage and exit.
  --version -V                    boolean  Displays pgloader version and exit.
  --quiet -q                      boolean  Be quiet
  --verbose -v                    boolean  Be verbose
  --debug -d                      boolean  Display debug level information.
  --client-min-messages           string   Filter logs seen at the console (default: "warning")
  --log-min-messages              string   Filter logs seen in the logfile (default: "notice")
  --summary -S                    string   Filename where to copy the summary
  --root-dir -D                   string   Output root directory. (default: #P"/tmp/pgloader/")
  --upgrade-config -U             boolean  Output the command(s) corresponding to .conf file for v2.x
  --list-encodings -E             boolean  List pgloader known encodings and exit.
  --logfile -L                    string   Filename where to send the logs.
  --load-lisp-file -l             string   Read user code from files
  --dry-run                       boolean  Only check database connections, don't load anything.
  --on-error-stop                 boolean  Refrain from handling errors properly.
  --no-ssl-cert-verification      boolean  Instruct OpenSSL to bypass verifying certificates.
  --context -C                    string   Command Context Variables
  --with                          string   Load options
  --set                           string   PostgreSQL options
  --field                         string   Source file fields specification
  --cast                          string   Specific cast rules
  --type                          string   Force input source type
  --encoding                      string   Source expected encoding
  --before                        string   SQL script to run before loading the data
  --after                         string   SQL script to run after loading the data
  --self-upgrade                  string   Path to pgloader newer sources
  --regress                       boolean  Drive regression testing

Usage

You can either give a command file to pgloader or run it all from the command line, see the pgloader quick start on https://pgloader.readthedocs.io for more details.

$ ./build/bin/pgloader --help
$ ./build/bin/pgloader <file.load>

For example, for a full migration from SQLite:

$ createdb newdb
$ pgloader ./test/sqlite/sqlite.db postgresql:///newdb

Or for a full migration from MySQL, including schema definition (tables, indexes, foreign keys, comments) and parallel loading of the corrected data:

$ createdb pagila
$ pgloader mysql://user@localhost/sakila postgresql:///pagila

LICENCE

pgloader is available under The PostgreSQL Licence.

INSTALL

Please see full documentation at https://pgloader.readthedocs.io/.

If you're using debian, it's already available:

$ apt-get install pgloader

If you're using docker, you can use the latest version built by the CI at each commit to the master branch:

$ docker pull ghcr.io/dimitri/pgloader:latest
$ docker run --rm -it ghcr.io/dimitri/pgloader:latest pgloader --version

atlas-engineer/nyxt

Nyxt - the hacker's browser.

Nyxt browser #+html:

Nyxt [nýkst] is a keyboard-driven web browser designed for hackers. Inspired by Emacs and Vim, it has familiar keybindings ([[https://en.wikipedia.org/wiki/Emacs][Emacs]], [[https://en.wikipedia.org/wiki/Vim_(text_editor)][vi]], [[https://en.wikipedia.org/wiki/IBM_Common_User_Access][CUA]]), and is infinitely extensible in Lisp.

Attention: Nyxt is under active development. Please feel free to [[https://github.com/atlas-engineer/nyxt/issues][report]] bugs, instabilities or feature wishes.

Features For an exhaustive description of all of the features, please refer to the manual.

** Fast tab switching

Switch easily between your open tabs via fuzzy search. If you are looking for ~https://www.example.com~, you could type in ~ele~, ~exa~, ~epl~, or any other matching series of letters.

#+html:

** Multiple marking

Commands can accept multiple inputs, allowing you to quickly perform an operation against multiple objects. In the example below we simultaneously open several bookmarks.

#+html:

** Powerful bookmarks

Bookmark a page with tags. Search bookmarks with compound queries. Capture more data about your bookmarks, and group and wrangle them in any way you like.

#+html:

** Multi tab search

Search multiple tabs at the same time, and view all the results in a single window. Jump quickly through your open tabs to find what you need.

#+html:

** History as a tree

History is represented as a tree that you can traverse. Smarter than the "forwards-backwards" abstraction found in other browsers, the tree makes sure you never lose track of where you've been.

#+html:

Installation

Supported platforms:

GNU/Linux
macOS (in development)
Windows (in development)
FreeBSD (unofficial)

** GNU/Linux

The Nyxt team maintains the following distribution means:

[[https://flathub.org/apps/engineer.atlas.Nyxt][Nyxt on Flathub]]

[[https://repology.org/project/nyxt/versions][Non-official distribution means are supported by the community as well]]. We're not accountable for their quality, so we kindly ask to report issues to the maintainers of those packaging efforts.

Contributing

Please refer to the [[file:developer-manual.org][developer's documentation]].