GitHub Scala Monthly Trending Repositories

apache/spark

Apache Spark - A unified analytics engine for large-scale data processing

https://spark.apache.org/

Apache Spark

Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R (Deprecated), and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.

Official version: https://spark.apache.org/
Development version: https://apache.github.io/spark/

Online Documentation

You can find the latest Spark documentation, including a programming guide, on the project web page. This README file only contains basic setup instructions.

Build Pipeline Status

Branch	Status
master

























branch-4.x








branch-4.3








branch-4.2








branch-4.1








branch-4.0







branch-3.5

Building Spark

Spark is built using Apache Maven. To build Spark and its example programs, run:

./build/mvn -DskipTests clean package

(You do not need to do this if you downloaded a pre-built package.)

More detailed documentation is available from the project site, at "Building Spark".

For general development tips, including info on developing Spark using an IDE, see "Useful Developer Tools".

Interactive Scala Shell

The easiest way to start using Spark is through the Scala shell:

./bin/spark-shell

Try the following command, which should return 1,000,000,000:

scala> spark.range(1000 * 1000 * 1000).count()

Interactive Python Shell

Alternatively, if you prefer Python, you can use the Python shell:

./bin/pyspark

And run the following command, which should also return 1,000,000,000:

>>> spark.range(1000 * 1000 * 1000).count()

Example Programs

Spark also comes with several sample programs in the examples directory. To run one of them, use ./bin/run-example <class> [params]. For example:

./bin/run-example SparkPi

will run the Pi example locally.

You can set the MASTER environment variable when running examples to submit examples to a cluster. This can be spark:// URL, "yarn" to run on YARN, and "local" to run locally with one thread, or "local[N]" to run locally with N threads. You can also use an abbreviated class name if the class is in the examples package. For instance:

MASTER=spark://host:7077 ./bin/run-example SparkPi

Many of the example programs print usage help if no params are given.

Running Tests

Testing first requires building Spark. Once Spark is built, tests can be run using:

./dev/run-tests

Please see the guidance on how to run tests for a module, or individual tests.

There is also a Kubernetes integration test, see resource-managers/kubernetes/integration-tests/README.md

A Note About Hadoop Versions

Spark uses the Hadoop core library to talk to HDFS and other Hadoop-supported storage systems. Because the protocols have changed in different versions of Hadoop, you must build Spark against the same version that your cluster runs.

Please refer to the build documentation at "Specifying the Hadoop Version and Enabling YARN" for detailed guidance on building for a particular distribution of Hadoop, including building for particular Hive and Hive Thriftserver distributions.

Configuration

Please refer to the Configuration Guide in the online documentation for an overview on how to configure Spark.

Contributing

Please review the Contribution to Spark guide for information on how to get started contributing to the project.

twitter/the-algorithm

Source code for the X Recommendation Algorithm

https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm

X's Recommendation Algorithm

X's Recommendation Algorithm is a set of services and jobs that are responsible for serving feeds of posts and other content across all X product surfaces (e.g. For You Timeline, Search, Explore, Notifications). For an introduction to how the algorithm works, please refer to our engineering blog.

Architecture

Product surfaces at X are built on a shared set of data, models, and software frameworks. The shared components included in this repository are listed below:

Type	Component	Description
Data	tweetypie	Core service that handles the reading and writing of post data.
	unified-user-actions	Real-time stream of user actions on X.
	user-signal-service	Centralized platform to retrieve explicit (e.g. likes, replies) and implicit (e.g. profile visits, tweet clicks) user signals.
Model	SimClusters	Community detection and sparse embeddings into those communities.
	TwHIN	Dense knowledge graph embeddings for Users and Posts.
	trust-and-safety-models	Models for detecting NSFW or abusive content.
	real-graph	Model to predict the likelihood of an X User interacting with another User.
	tweepcred	Page-Rank algorithm for calculating X User reputation.
	recos-injector	Streaming event processor for building input streams for GraphJet based services.
	graph-feature-service	Serves graph features for a directed pair of users (e.g. how many of User A's following liked posts from User B).
	topic-social-proof	Identifies topics related to individual posts.
	representation-scorer	Compute scores between pairs of entities (Users, Posts, etc.) using embedding similarity.
Software framework	navi	High performance, machine learning model serving written in Rust.
	product-mixer	Software framework for building feeds of content.
	timelines-aggregation-framework	Framework for generating aggregate features in batch or real time.
	representation-manager	Service to retrieve embeddings (i.e. SimClusers and TwHIN).
	twml	Legacy machine learning framework built on TensorFlow v1.

The product surfaces currently included in this repository are the For You Timeline and Recommended Notifications.

For You Timeline

The diagram below illustrates how major services and jobs interconnect to construct a For You Timeline.

The core components of the For You Timeline included in this repository are listed below:

Type	Component	Description
Candidate Source	search-index	Find and rank In-Network posts. ~50% of posts come from this candidate source.
	tweet-mixer	Coordination layer for fetching Out-of-Network tweet candidates from underlying compute services.
	user-tweet-entity-graph (UTEG)	Maintains an in memory User to Post interaction graph, and finds candidates based on traversals of this graph. This is built on the GraphJet framework. Several other GraphJet based features and candidate sources are located here.
	follow-recommendation-service (FRS)	Provides Users with recommendations for accounts to follow, and posts from those accounts.
Ranking	light-ranker	Light Ranker model used by search index (Earlybird) to rank posts.
	heavy-ranker	Neural network for ranking candidate posts. One of the main signals used to select timeline posts post candidate sourcing.
Post mixing & filtering	home-mixer	Main service used to construct and serve the Home Timeline. Built on product-mixer.
	visibility-filters	Responsible for filtering X content to support legal compliance, improve product quality, increase user trust, protect revenue through the use of hard-filtering, visible product treatments, and coarse-grained downranking.
	timelineranker	Legacy service which provides relevance-scored posts from the Earlybird Search Index and UTEG service.

Recommended Notifications

The core components of Recommended Notifications included in this repository are listed below:

Type	Component	Description
Service	pushservice	Main recommendation service at X used to surface recommendations to our users via notifications.
Ranking	pushservice-light-ranker	Light Ranker model used by pushservice to rank posts. Bridges candidate generation and heavy ranking by pre-selecting highly-relevant candidates from the initial huge candidate pool.
	pushservice-heavy-ranker	Multi-task learning model to predict the probabilities that the target users will open and engage with the sent notifications.

Build and test code

We include Bazel BUILD files for most components, but not a top-level BUILD or WORKSPACE file. We plan to add a more complete build and test system in the future.

Contributing

We invite the community to submit GitHub issues and pull requests for suggestions on improving the recommendation algorithm. We are working on tools to manage these suggestions and sync changes to our internal repository. Any security concerns or issues should be routed to our official bug bounty program through HackerOne. We hope to benefit from the collective intelligence and expertise of the global community in helping us identify issues and suggest improvements, ultimately leading to a better X.

Read our blog on the open source initiative here.

lichess-org/lila

♞ lichess.org: the forever free, adless and open source chess server ♞

https://lichess.org

lichess.org

Lila (li[chess in sca]la) is a free online chess game server focused on realtime gameplay and ease of use.

It features a search engine, computer analysis distributed with fishnet, tournaments, simuls, forums, teams, tactic trainer, a mobile app, and a shared analysis board. The UI is available in more than 140 languages thanks to the community.

Lichess is written in Scala 3, and relies on a modified Play 2.8 framework. scalatags is used for templating. Pure chess logic is contained in the scalachess submodule. The server is fully asynchronous, making heavy use of Scala Futures and Pekko streams. WebSocket connections are handled by a separate server that communicates using redis. Lichess talks to Stockfish deployed in an AI cluster of donated servers. It uses MongoDB to store more than 12 billion games, which are indexed by elasticsearch. HTTP requests and WebSocket connections can be proxied by nginx. The web client is written in TypeScript and snabbdom, using Sass to generate CSS. All rated games are published in a free PGN database. Browser testing done with Browserstack. Proxy detection done with IP2Proxy database. Please help us translate Lichess with Crowdin.

See lichess.org/source for a list of repositories.

Join us on Discord for more info. Use GitHub issues for bug reports and feature requests.

Installation

./lila.sh # thin wrapper around sbt
run

The Wiki describes how to setup a development environment.

HTTP API

Feel free to use the Lichess API in your applications and websites.

Supported browsers

Name	Version	Notes
Firefox	115+	Full support (recommended)
Chromium / Chrome	112+	Full support
Edge	111+	Full support
Opera	97+	Reasonable support
Safari	16.2+	Reasonable support

Older browsers will not work. For your own sake, please upgrade. Security and performance, think about it!

License

Lila is licensed under the GNU Affero General Public License 3 or any later version at your choice. See copying for details.

Production architecture (as of July 2022)

Credits

See lichess.org/thanks and the contributors here:

Competence development program

Lichess would like to support its contributors in their competence development by covering costs of relevant training materials and activities. This is a small way to further empower contributors who have given their time to Lichess and to enable or improve additional contributions to Lichess in the future. For more information, including how to apply, check Competence Development for Lichess contributors.

joernio/joern

Open-source code analysis platform for C/C++/Java/Binary/Javascript/Python/Kotlin based on code property graphs. Discord https://discord.gg/vv4MH284Hc

https://joern.io/

Joern - The Bug Hunter's Workbench

Joern is a platform for analyzing source code, bytecode, and binary executables. It generates code property graphs (CPGs), a graph representation of code for cross-language code analysis. Code property graphs are stored in a custom graph database. This allows code to be mined using search queries formulated in a Scala-based domain-specific query language. Joern is developed with the goal of providing a useful tool for vulnerability discovery and research in static program analysis.

Website: https://joern.io

Documentation: https://docs.joern.io/

Specification: https://cpg.joern.io

News / Changelog

Joern v4.0.0 migrates from overflowdb to flatgraph
Joern v2.0.0 upgrades from Scala2 to Scala3
Joern v1.2.0 removes the overflowdb.traversal.Traversal class. This change is not completely backwards compatible. See here for a detailed writeup.

Requirements

JDK 21 (other versions might work, but have not been properly tested)
optional: gcc and g++ (for auto-discovery of C/C++ system header files if included/used in your C/C++ code)

Quick Installation

wget https://github.com/joernio/joern/releases/latest/download/joern-install.sh
chmod +x ./joern-install.sh
sudo ./joern-install.sh
joern

     ██╗ ██████╗ ███████╗██████╗ ███╗   ██╗
     ██║██╔═══██╗██╔════╝██╔══██╗████╗  ██║
     ██║██║   ██║█████╗  ██████╔╝██╔██╗ ██║
██   ██║██║   ██║██╔══╝  ██╔══██╗██║╚██╗██║
╚█████╔╝╚██████╔╝███████╗██║  ██║██║ ╚████║
 ╚════╝  ╚═════╝ ╚══════╝╚═╝  ╚═╝╚═╝  ╚═══╝
Version: 2.0.1
Type `help` to begin

joern>

If the installation script fails for any reason, try

./joern-install --interactive

Development Requirements

java
sbt

Run unit and integration tests locally

Unit tests:

sbt test

Integration tests:

sbt joerncli/stage querydb/createDistribution
python -m pip install requests pexpect # wexpect on Windows
python -u ./testDistro.py

There is an experimental partial Bazel build setup. Check bazel.md for further details.

Docker based execution

docker run --rm -it -v /tmp:/tmp -v $(pwd):/app:rw -w /app -t ghcr.io/joernio/joern joern

To run joern in server mode:

docker run --rm -it -v /tmp:/tmp -v $(pwd):/app:rw -w /app -t ghcr.io/joernio/joern joern --server

Almalinux 9 requires the CPU to support SSE4.2. For kvm64 VM use the Almalinux 8 version instead.

docker run --rm -it -v /tmp:/tmp -v $(pwd):/app:rw -w /app -t ghcr.io/joernio/joern-alma8 joern

Releases

A new release is created automatically once per day. Contributers can also manually run the release workflow if they need the release sooner.

Each release publishes platform-specific distribution zips:

joern-cli-linux-x86_64.zip
joern-cli-linux-arm64.zip
joern-cli-macos-x86_64.zip
joern-cli-macos-arm64.zip
joern-cli-windows-x86_64.zip
joern-cli-windows-arm64.zip

joern-install.sh detects the current platform automatically and downloads the correct zip. Windows users should download the appropriate zip directly from the releases page and extract it manually.

Developers

Contribution Guidelines

Thank you for taking time to contribute to Joern! Here are a few guidelines to ensure your pull request will get merged as soon as possible:

Try to make use of the templates as far as possible, however they may not suit all needs. The minimum we would like to see is:
- A title that briefly describes the change and purpose of the PR, preferably with the affected module in square brackets, e.g. [javasrc2cpg] Addition Operator Fix.
- A short description of the changes in the body of the PR. This could be in bullet points or paragraphs.
- A link or reference to the related issue, if any exists.
Do not:
- Immediately CC/@/email spam other contributors, the team will review the PR and assign the most appropriate contributor to review the PR. Joern is maintained by industry partners and researchers alike, for the most part with their own goals and priorities, and additional help is largely volunteer work. If your PR is going stale, then reach out to us in follow-up comments with @'s asking for an explanation of priority or planning of when it may be addressed (if ever, depending on quality).
- Leave the description body empty, this makes reviewing the purpose of the PR difficult.
Remember to:
- Remember to format your code, i.e. run sbt scalafmt Test/scalafmt
- Add a unit test to verify your change.

IDE setup

Intellij IDEA

Download Intellij Community
Install and run it
Install the Scala Plugin - just search and install from within Intellij.
Important: open sbt in your local joern repository, run compile and keep it open - this will allow us to use the BSP build in the next step
Back to Intellij: open project: select your local joern clone: select to open as BSP project (i.e. not sbt project!)
Await the import and indexing to complete, then you can start, e.g. Build -> build project or run a test

VSCode

Install VSCode and Docker
Install the plugin ms-vscode-remote.remote-containers
Open Joern project folder in VSCode
- Option 1: Visual Studio Code detects the new files and opens a message box saying: Folder contains a Dev Container configuration file. Reopen to folder to develop in a container.. Select the Reopen in Container button to reopen the folder in the container created by the .devcontainer/Dockerfile file.
- Option 2: press Ctrl + Shift + P then select Dev Containers: Reopen in Container
Press Ctrl + Shift + P then select Metals: Import build
After Metals: Import build succeeds, you are ready to start writing code for Joern

QueryDB (queries plugin)

Quick way to develop and test QueryDB:

sbt stage
./querydb-install.sh
./joern-scan --list-query-names

The last command prints all available queries - add your own in querydb, run the above commands again to see that your query got deployed. More details in the separate querydb readme

scala/scala

Scala 2 compiler and standard library. Scala 2 bugs at https://github.com/scala/bug; Scala 3 at https://github.com/scala/scala3

https://www.scala-lang.org/

This is Scala 2! Welcome!

This is the home of the Scala 2 standard library, compiler, and language spec.

For Scala 3, visit scala/scala3.

How to contribute

Issues and bug reports for Scala 2 are located in scala/bug. That tracker is also where new contributors may find issues to work on: good first issues, help wanted.

For coordinating broader efforts, we also use the scala/scala-dev tracker.

To contribute here, please open a pull request from your fork of this repository.

If your change affects the standard library, see CONTRIBUTING.md for remarks about what sort of changes might be accepted in Scala 2 and/or Scala 3.

We require that you sign the Scala CLA before we can merge any of your work, to protect Scala's future as open source software.

The general workflow is as follows.

Find/file an issue in scala/bug (or submit a well-documented PR right away!).
Fork the scala/scala repo.
Push your changes to a branch in your forked repo. For coding guidelines, go here.
Submit a pull request to scala/scala from your forked repo.

For more information on building and developing the core of Scala, read the rest of this README, especially for setting up your machine!

Get in touch!

In order to get in touch with other Scala contributors, join the #scala-contributors channel on the Scala Discord chat, or post on contributors.scala-lang.org (Discourse).

If you need some help with your PR at any time, please feel free to @-mention anyone from the list below, and we will do our best to help you out:

	username	talk to me about...
	`@lrytz`	back end, optimizer, named & default arguments, reporters
	`@retronym`	compiler performance, weird compiler bugs, lambdas
	`@SethTisue`	getting started, build, CI, community build, Jenkins, docs, library, REPL
	`@dwijnand`	pattern matcher, MiMa, partest
	`@som-snytt`	warnings/lints/errors, REPL, compiler options, compiler internals, partest
	`@Ichoran`	collections library, performance
	`@viktorklang`	concurrency, futures
	`@sjrd`	interactions with Scala.js
	`@bishabosha`	TASTy reader
	`@joroKr21`	higher-kinded types, implicits, variance

P.S.: If you have some spare time to help out around here, we would be delighted to add your name to this list!

Branches

Target the oldest branch you would like your changes to end up in. We periodically merge forward from 2.12.x to 2.13.x. Most changes should target 2.13.x, as 2.12.x is now under minimal maintenance.

If your change is difficult to merge forward, you may be asked to also submit a separate PR targeting the newer branch.

If your change is version-specific and shouldn't be merged forward, put [nomerge] in the PR name.

If your change is a backport from a newer branch and thus doesn't need to be merged forward, put [backport] in the PR name.

Choosing a branch

Most changes should target 2.13.x. We are increasingly reluctant to target 2.12.x unless there is a special reason (e.g. if an especially bad bug is found, or if there is commercial sponsorship). See Scala 2 maintenance.

Repository structure

Most importantly:

scala/
+--build.sbt                 The main sbt build definition
+--project/                  The rest of the sbt build
+--src/                      All sources
   +---/library              Scala Standard Library
   +---/reflect              Scala Reflection
   +---/compiler             Scala Compiler
+--test/                     The Scala test suite
   +---/files                Partest tests
   +---/junit                JUnit tests
   +---/scalacheck           ScalaCheck tests
+--spec/                     The Scala language specification

but also:

scala/
   +---/library-aux          Scala Auxiliary Library, for bootstrapping and documentation purposes
   +---/interactive          Scala Interactive Compiler, for clients such as an IDE (aka Presentation Compiler)
   +---/manual               Scala's runner scripts "man" (manual) pages
   +---/partest              Scala's internal parallel testing framework
   +---/partest-javaagent    Partest's helper java agent
   +---/repl                 Scala REPL core
   +---/repl-frontend        Scala REPL frontend
   +---/scaladoc             Scala's documentation tool
   +---/scalap               Scala's class file decompiler
   +---/testkit              Scala's unit-testing kit
+--admin/                    Scripts for the CI jobs and releasing
+--doc/                      Additional licenses and copyrights
+--scripts/                  Scripts for the CI jobs and releasing
+--tools/                    Scripts useful for local development
+--build/                    Build products
+--dist/                     Build products
+--target/                   Build products

Get ready to contribute

Requirements

You need the following tools:

Java SDK. The baseline version is 8 for both 2.12.x and 2.13.x. It is almost always fine to use a later SDK (such as 17 or 21) for local development. CI will verify against the baseline version.
sbt

MacOS and Linux work. Windows may work if you use Cygwin. Community help with keeping the build working on Windows and documenting any needed setup is appreciated.

Tools we use

We are grateful for the following OSS licenses:

Build setup

Basics

During ordinary development, a new Scala build is built by the previously released version, known as the "reference compiler" or, slangily, as "STARR" (stable reference release). Building with STARR is sufficient for most kinds of changes.

However, a full build of Scala is bootstrapped. Bootstrapping has two steps: first, build with STARR; then, build again using the freshly built compiler, leaving STARR behind. This guarantees that every Scala version can build itself.

If you change the code generation part of the Scala compiler, your changes will only show up in the bytecode of the library and compiler after a bootstrap. Our CI does a bootstrapped build.

Bootstrapping locally: To perform a bootstrap, run restarrFull within an sbt session. This will build and publish the Scala distribution to your local artifact repository and then switch sbt to use that version as its new scalaVersion. You may then revert back with reload. Note restarrFull will also write the STARR version to buildcharacter.properties so you can switch back to it with restarr without republishing. This will switch the sbt session to use the build-restarr and target-restarr directories instead of build and target, which avoids wiping out classfiles and incremental metadata. IntelliJ will continue to be configured to compile and run tests using the starr version in versions.properties.

For history on how the current scheme was arrived at, see https://groups.google.com/d/topic/scala-internals/gp5JsM1E0Fo/discussion.

Building with fatal warnings: To make warnings in the project fatal (i.e. turn them into errors), run set Global / fatalWarnings := true in sbt (replace Global with the name of a module—such as reflect—to only make warnings fatal for that module). To disable fatal warnings again, either reload sbt, or run set Global / fatalWarnings := false (again, replace Global with the name of a module if you only enabled fatal warnings for that module). CI always has fatal warnings enabled.

Using the sbt build

Once you've started an sbt session you can run one of the core commands:

compile compiles all sub-projects (library, reflect, compiler, scaladoc, etc)
scala / scalac run the REPL / compiler directly from sbt (accept options / arguments)
enableOptimizer reloads the build with the Scala optimizer enabled. Our releases are built this way. Enable this when working on compiler performance improvements. When the optimizer is enabled the build will be slower and incremental builds can be incorrect.
setupPublishCore runs enableOptimizer and configures a version number based on the current Git SHA. Often used as part of bootstrapping: sbt setupPublishCore publishLocal && sbt -Dstarr.version=<VERSION> testAll
dist/mkBin generates runner scripts (scala, scalac, etc) in build/quick/bin
dist/mkPack creates a build in the Scala distribution format in build/pack
junit/test runs the JUnit tests; junit/testOnly *Foo runs a subset
scalacheck/test runs scalacheck tests, use testOnly to run a subset
partest runs partest tests (accepts options, try partest --help)
publishLocal publishes a distribution locally (can be used as scalaVersion in other sbt projects)
- Optionally set baseVersionSuffix := "bin-abcd123-SNAPSHOT" where abcd123 is the git hash of the revision being published. You can also use something custom like "bin-mypatch". This changes the version number from 2.13.2-SNAPSHOT to something more stable (2.13.2-bin-abcd123-SNAPSHOT).
- Note that the -bin string marks the version binary compatible. Using it in sbt will cause the scalaBinaryVersion to be 2.13. If the version is not binary compatible, we recommend using -pre, e.g., 2.14.0-pre-abcd123-SNAPSHOT.
- Optionally set ThisBuild / Compile / packageDoc / publishArtifact := false to skip generating / publishing API docs (speeds up the process).

If a command results in an error message like a module is not authorized to depend on itself, it may be that a global sbt plugin is causing a cyclical dependency. Try disabling global sbt plugins (perhaps by temporarily commenting them out in ~/.sbt/1.0/plugins/plugins.sbt).

Sandbox

We recommend keeping local test files in the sandbox directory which is listed in the .gitignore of the Scala repo.

IDE setup

In IntelliJ IDEA, use "File - Open...", select the project folder and use "Open as: sbt project".

JUnit tests can be launched / debugged from the IDE, including tests that run the compiler (e.g., QuickfixTest).
Building in IntelliJ interoperates with the sbt build; the compiler script in build/quick/bin (generated by running sbt dist/mkBin) runs the classfiles built via the IDE.

In VSCode / Metals, open the project directory and import the project as ususal.

Coding guidelines

Our guidelines for contributing are explained in CONTRIBUTING.md. It contains useful information on our coding standards, testing, documentation, how we use git and GitHub and how to get your code reviewed.

Refer to our LLM policy for rules and guidelines regarding the use of LLM-based tools in your contributions.

You may also want to check out the following resources:

The "Scala Hacker Guide" covers some of the same ground as this README, but in greater detail and in a more tutorial style, using a running example.
Scala documentation site

Scala CI

Once you submit a PR your commits will be automatically tested by the Scala CI.

Our CI setup is always evolving. See scala/scala-dev#751 for more details on how things currently work and how we expect they might change.

If you see a spurious failure on Jenkins, you can post /rebuild as a PR comment. The scabot README lists all available commands.

If you'd like to test your patch before having everything polished for review, you can have Travis CI build your branch (make sure you have a fork and have Travis CI enabled for branch builds on it first, and then push your branch). Also feel free to submit a draft PR. In case your draft branch contains a large number of commits (that you didn't clean up / squash yet for review), consider adding [ci: last-only] to the PR title. That way only the last commit will be tested, saving some energy and CI-resources. Note that inactive draft PRs will be closed eventually, which does not mean the change is being rejected.

CI performs a compiler bootstrap. The first task, validatePublishCore, publishes a build of your commit to the temporary repository https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots. Note that this build is not yet bootstrapped, its bytecode is built using the current STARR. The version number is 2.13.2-bin-abcd123-SNAPSHOT where abcd123 is the commit hash. For binary incompatible builds, the version number is 2.14.0-pre-abcd123-SNAPSHOT.

You can use Scala builds in the validation repository locally by adding a resolver and specifying the corresponding scalaVersion:

$ sbt
> set resolvers += "pr" at "https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/"
> set scalaVersion := "2.13.17-bin-abcd123-SNAPSHOT"
> console

"Nightly" builds

The Scala CI publishes these to https://scala-ci.typesafe.com/artifactory/scala-integration/ .

Using a nightly build in sbt and other tools is explained on this doc page.

Although we casually refer to these as "nightly" builds, they aren't actually built nightly, but "mergely". That is to say, a build is published for every merged PR.

Scala CI internals

The Scala CI runs as a Jenkins instance on scala-ci.typesafe.com, configured by a chef cookbook at scala/scala-jenkins-infra.

The build bot that watches PRs, triggers testing builds and applies the "reviewed" label after an LGTM comment is in the scala/scabot repo.

Community build

The Scala community build is an important method for testing Scala releases. A community build can be launched for any Scala commit, even before the commit's PR has been merged. That commit is then used to build a large number of open-source projects from source and run their test suites.

To request a community build run on your PR, just ask in a comment on the PR and a Scala team member (probably @SethTisue) will take care of it. (details)

Community builds run on the Scala Jenkins instance. The jobs are named ..-integrate-community-build. See the scala/community-builds repo.

apache/gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.

https://gluten.apache.org/

A Middle Layer for Offloading JVM-Based SQL Execution to Native Engines

1. Introduction

Background

Apache Spark is a mature and stable project that has been under continuous development for many years. It is one of the most widely used frameworks for scaling out the processing of petabyte-scale datasets. Over time, the Spark community has had to address significant performance challenges, which required a variety of optimizations. A major milestone came with Spark 2.0, where Whole-Stage Code Generation replaced the Volcano Model, delivering up to a 2× speedup. Since then, most subsequent improvements have focused on the query plan level, while the performance of individual operators has almost stopped improving.

In recent years, several native SQL engines have been developed, such as ClickHouse and Velox. With features like native execution, columnar data formats, and vectorized data processing, these engines can outperform Spark’s JVM-based SQL engine. However, they currently don't directly support Spark SQL execution.

Design Overview

“Gluten” is Latin for "glue". The main goal of the Gluten project is to glue native engines to Spark SQL. Thus, we can benefit from the high performance of native engines and the high scalability enabled by the Spark ecosystem.

The basic design principle is to reuse Spark’s control flow, while offloading compute-intensive data processing to the native side. More specifically:

Transform Spark’s physical plan into a Substrait plan, then transform it into the native engine's plan.
Offload performance-critical data processing to native engine.
Define clear JNI interfaces for native SQL engines.
Allow easy switching between available native backends.
Reuse Spark’s distributed control flow.
Manage data sharing between JVM and native.
Provide extensibility to support more native engines.

Target Users

Gluten's target users include anyone who wants to fundamentally accelerate Spark SQL. As a plugin to Spark, Gluten requires no changes to the DataFrame API or SQL queries; users only need to configure it correctly.

2. Architecture

The overview chart is shown below. Substrait provides a well-defined, cross-language specification for data compute operations. Spark’s physical plan is transformed into a Substrait plan, which is then passed to the native side through a JNI call. On the native side, a chain of native operators is constructed and offloaded to the native engine. Gluten returns the results as a ColumnarBatch, and Spark’s Columnar API (introduced in Spark 3.0) is used during execution. Gluten adopts the Apache Arrow data format as its underlying representation.

Currently, Gluten supports only the ClickHouse and Velox backends. Velox is a C++ database acceleration library that provides reusable, extensible, and high-performance data processing components. In addition, Gluten is designed to be extensible, allowing support for additional backends in the future.

Gluten's key components:

Query Plan Conversion: Converts Spark's physical plan to Substrait plan.
Unified Memory Management: Manages native memory allocation.
Columnar Shuffle: Handles shuffling of Gluten's columnar data. The shuffle service of Spark core is reused, while a columnar exchange operator is implemented to support Gluten's columnar data format.
Fallback Mechanism: Provides fallback to vanilla Spark for unsupported operators. Gluten's ColumnarToRow (C2R) and RowToColumnar (R2C) convert data between Gluten's columnar format and Spark's internal row format to support fallback transitions.
Metrics: Collected from Gluten native engine to help monitor execution, identify bugs, and diagnose performance bottlenecks. The metrics are displayed in Spark UI.
Shim Layer: Ensures compatibility with multiple Spark versions. Gluten supports the latest 3–4 Spark releases during its development cycle, and currently supports Spark 3.3, 3.4, 3.5, 4.0, and 4.1.

3. User Guide

Below is a basic configuration to enable Gluten in Spark.

export GLUTEN_JAR=/PATH/TO/GLUTEN_JAR
spark-shell \
  --master yarn --deploy-mode client \
  --conf spark.plugins=org.apache.gluten.GlutenPlugin \
  --conf spark.memory.offHeap.enabled=true \
  --conf spark.memory.offHeap.size=20g \
  --conf spark.driver.extraClassPath=${GLUTEN_JAR} \
  --conf spark.executor.extraClassPath=${GLUTEN_JAR} \
  --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
  ...

There are two ways to acquire Gluten jar for the above configuration.

Use Released JAR

Please download the tar package here, then extract the Gluten JAR from it. Additionally, Gluten provides nightly builds based on the main branch for early testing. The nightly build JARs are available at Apache Gluten Nightlies. They have been verified on Centos 7/8/9, Ubuntu 20.04/22.04.

Build From Source

For Velox backend, please refer to Velox.md and build-guide.md.

For ClickHouse backend, please refer to ClickHouse.md.

The Gluten JAR will be generated under /PATH/TO/GLUTEN/package/target/ after the build.

Configurations

Common configurations used by Gluten are listed in Configuration.md. Velox specific configurations are listed in velox-configuration.md.

The Gluten Velox backend honors some Spark configurations, ignores others, and many are transparent to it. See velox-spark-configuration.md for details, and velox-parquet-write-configuration.md for Parquet write configurations.

4. Resources

5. Contribution

Welcome to contribute to the Gluten project! See CONTRIBUTING.md for guidelines on how to make contributions.

6. Community

Here are several ways to connect with the community.

GitHub

Welcome to report issues or start discussions in GitHub. Please search the GitHub issue list before creating a new one to avoid duplication.

Mailing List

For any technical discussions, please email dev@gluten.apache.org. You can browse the archives to view past discussions, or subscribe to the mailing list to receive updates.

Slack Channel (English)

Request an invitation to the ASF Slack workspace via this page. Once invited, you can join the gluten channel.

The ASF Slack login entry: https://the-asf.slack.com/.

WeChat Group (Chinese)

Please contact weitingchen at apache.org or zhangzc at apache.org to request an invitation to the WeChat group. It is for Chinese-language communication.

7. Performance

TPC-H is used to evaluate Gluten's performance. Please note that the results below do not reflect the latest performance.

Velox Backend

The Gluten Velox backend demonstrated an overall speedup of 2.71x, with up to a 14.53x speedup observed in a single query.

_{Tested in Jun. 2023. Test environment: single node with 2TB data, using Spark 3.3.2 as the baseline and with Gluten integrated into the same Spark version.}

ClickHouse Backend

ClickHouse backend demonstrated an average speedup of 2.12x, with up to 3.48x speedup observed in a single query.

_{Test environment: a 8-nodes AWS cluster with 1TB data, using Spark 3.1.1 as the baseline and with Gluten integrated into the same Spark version.}

8. Qualification Tool

The Qualification Tool is a utility to analyze Spark event log files and assess the compatibility and performance of SQL workloads with Gluten. This tool helps users understand how their workloads can benefit from Gluten.

9. License

Gluten is licensed under Apache License Version 2.0.

10. Acknowledgements

Gluten was initiated by Intel and Kyligence in 2022 and became an Apache Top-Level Project in March 2026. Several other companies are also actively contributing to its development, including BIGO, Meituan, Alibaba Cloud, NetEase, Baidu, Microsoft, IBM, Google, and others.

OpenXiangShan/XiangShan

Open-source high-performance RISC-V processor

https://xiangshan.cc

XiangShan

XiangShan (香山) is an open-source high-performance RISC-V processor project.

English | 中文

Documentation

XiangShan's documentation is available at docs.xiangshan.cc.

XiangShan Design Document for Kunminghu V2R2 has been published separately. You can find it at docs.xiangshan.cc/projects/design.

XiangShan User Guide has been published separately. You can find it at docs.xiangshan.cc/projects/user-guide or XiangShan-User-Guide/releases.

We are using Weblate to translate documentation into English and other languages. Your contributions are welcome—come and help us improve it!

All XiangShan documents are licensed under the CC-BY-4.0.

Publications

MICRO 2022: Towards Developing High Performance RISC-V Processors Using Agile Methodology

Our paper introduces XiangShan and the practice of agile development methodology on high performance RISC-V processors. It covers some representative tools we have developed and used to accelerate the chip development process, including design, functional verification, debugging, performance validation, etc. This paper is awarded all three available badges for artifact evaluation (Available, Functional, and Reproduced).

Paper PDF | IEEE Xplore | BibTeX | Presentation Slides | Presentation Video

Wechat/微信：香山开源处理器

Zhihu/知乎：香山开源处理器

Weibo/微博：香山开源处理器

You can contact us through our mailing list. All mails from this list will be archived here.

Architecture

The first stable micro-architecture of XiangShan is called Yanqihu (雁栖湖) and is on the yanqihu branch, which has been developed since June 2020.

The second stable micro-architecture of XiangShan is called Nanhu (南湖) and is on the nanhu branch.

The third-generation XiangShan microarchitecture, Kunminghu (昆明湖), is under active development on the kunminghu-v2 and kunminghu-v3 branches.

The micro-architecture overview of Kunminghu-V2 is shown below.

Branch Maintenance Status

Generation-Version	Branch	Maintained?	Active Development?
Yanqihu	yanqihu	✅ (minimum)
Nanhu-V2	nanhu	✅ (minimum)
Nanhu-V3	Siudya/Nanhu
Nanhu-V5	OpenXiangShan-Nanhu/OpenNanhu-V5	✅	✅
Kunminghu-V2	kunminghu-v2	✅
Kunminghu-V3	kunminghu-v3	✅	✅

At the moment, kunminghu-v3 is still evolving rapidly and its functionality may not yet be stable. If you plan to use XiangShan for research, verification, or downstream applications, we recommend prioritizing kunminghu-v2.

This section is time-sensitive. If you believe any information here is outdated, please let us know in an issue and we will update it as soon as possible.

Last updated: 2026/06/30

Sub-directories Overview

Some of the key directories are shown below.

.
├── src
│   └── main/scala         # design files
│       ├── device         # virtual device for simulation
│       ├── system         # SoC wrapper
│       ├── top            # top module
│       ├── utils          # utilization code
│       └── xiangshan      # main design code
│           └── transforms # some useful firrtl transforms
├── scripts                # scripts for agile development
├── yunsuan                # yunsuan submodule of XiangShan
├── XSCache                # cache subsystem of XiangShan
├── difftest               # difftest co-simulation framework
└── ready-to-run           # pre-built simulation images

IDE Support

bsp

make bsp

IDEA

make idea

Generate Verilog

Run make verilog to generate verilog code. This generates multiple .sv files in the build/rtl/ folder (e.g., build/rtl/XSTop.sv).
Refer to Makefile for more information.

Run Programs by Simulation

Prepare environment

Set environment variable NEMU_HOME to the absolute path of the NEMU project.
Set environment variable NOOP_HOME to the absolute path of the XiangShan project.
Set environment variable AM_HOME to the absolute path of the AM project.
Install mill. Refer to the Manual section in this guide.
Clone this project and run make init to initialize submodules.

Run with simulator

Install Verilator, the open-source Verilog simulator.
Run make emu to build the C++ simulator ./build/emu with Verilator.
Refer to ./build/emu --help for run-time arguments of the simulator.
Refer to Makefile and verilator.mk for more information.

Example:

make emu CONFIG=MinimalConfig EMU_THREADS=2 -j10
./build/emu -b 0 -e 0 -i ./ready-to-run/coremark-2-iteration.bin --diff ./ready-to-run/riscv64-nemu-interpreter-so

Run with xspdb

There are two ways to use xspdb:

Quick Start with Prebuilt Binaries
- Why choose this? It is lightweight and requires no compilation. A standard Python environment is enough to run the full XiangShan experience with low memory usage.
- Step 1: Download Get the latest XSPdb from the run summary of the repository's Actions workflow.
- Step 2: Extract & Run
Build from source code
- Install picker, a verification tool that supports high-level languages.
- Run make pdb to build XiangShan Python binaries.
- Run make pdb-run to run XiangShan binaries.

Example output and interaction:

$ make pdb-run
[Info] Set PMEM_BASE to 0x80000000 (Current: 0x80000000)
[Info] Set FIRST_INST_ADDRESS to 0x80000000 (Current: 0x80000000)
Using simulated 32768B flash
[Info] reset dut complete
> XiangShan/scripts/pdb-run.py(13)run()
-> while True:
(XiangShan) xload ready-to-run/microbench.bin   # Load binary (Tab-compatible)
(XiangShan) xwatch_commit_pc 0x80000004         # Set watchpoint
(XiangShan) xistep 3                            # Step until the next three instructions commit; stops at the watchpoint if hit
[Info] Find break point (Inst commit), break (step 2107 cycles) at cycle: 2207 (0x89f)
[Info] Find break point (Inst commit, Target commit), break (step 2108 cycles) at cycle: 2208 (0x8a0)
(XiangShan) xpc                                 # Print PC info
PC[0]: 0x80000000    Instr: 0x00000093
PC[1]: 0x80000004    Instr: 0x00000113
PC[2]: 0x0    Instr: 0x0
...
PC[7]: 0x0    Instr: 0x0
(XiangShan) xistep 1000000                      # Execute until the end of the binary
[Info] Find break point (Inst commit), break (step 2037 cycles) at cycle: 2207 (0x89f)
[Info] Find break point (Inst commit), break (step 2180 cycles) at cycle: 2207 (0x89f)
...
HIT GOOD LOOP at pc = 0xf0001cb0

Troubleshooting Guide

Acknowledgement

The implementation of XiangShan is inspired by several key papers. We list these papers in XiangShan document, see: Acknowledgements. We very much encourage and expect that more academic innovations can be realised based on XiangShan in the future.

LICENSE

XiangShan is licensed under Mulan PSL v2.

scala/scala3

The Scala 3 compiler, also known as Dotty.

https://scala-lang.org

Scala 3

This is the home of the Scala 3 standard library, compiler, and language spec. More documentation here.

Building a Local Distribution

sbt dist/Universal/packageBin
Find the newly-built distributions in dist/target/

Code of Conduct

Scala 3 uses the Scala Code of Conduct for all communication and discussion. This includes both GitHub, Discord and other more direct lines of communication such as email.

How to Contribute

License

Scala 3 is licensed under the Apache License Version 2.0

delta-io/delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Delta Lake is an open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs for Scala, Java, Rust, Ruby, and Python.

See the Delta Lake Documentation for details.
See the Quick Start Guide to get started with Scala, Java and Python.
Note, this repo is one of many Delta Lake repositories in the delta.io organizations including delta, delta-rs, delta-sharing, kafka-delta-ingest, and website.

The following are some of the more popular Delta Lake integrations, refer to delta.io/integrations for the complete list:

Apache Spark™: This connector allows Apache Spark™ to read from and write to Delta Lake.
Apache Flink (Preview): This connector allows Apache Flink to write to Delta Lake.
PrestoDB: This connector allows PrestoDB to read from Delta Lake.
Trino: This connector allows Trino to read from and write to Delta Lake.
Delta Standalone: This library allows Scala and Java-based projects (including Apache Flink, Apache Hive, Apache Beam, and PrestoDB) to read from and write to Delta Lake.
Apache Hive: This connector allows Apache Hive to read from Delta Lake.
Delta Rust API: This library allows Rust (with Python and Ruby bindings) low level access to Delta tables and is intended to be used with data processing frameworks like datafusion, ballista, rust-dataframe, vega, etc.

Table of Contents

Latest Binaries

See the online documentation for the latest release.

API Documentation

Compatibility

Delta Standalone library is a single-node Java library that can be used to read from and write to Delta tables. Specifically, this library provides APIs to interact with a table’s metadata in the transaction log, implementing the Delta Transaction Log Protocol to achieve the transactional guarantees of the Delta Lake format.

API Compatibility

There are two types of APIs provided by the Delta Lake project.

Direct Java/Scala/Python APIs - The classes and methods documented in the API docs are considered as stable public APIs. All other classes, interfaces, methods that may be directly accessible in code are considered internal, and they are subject to change across releases.
Spark-based APIs - You can read Delta tables through the DataFrameReader/Writer (i.e. spark.read, df.write, spark.readStream and df.writeStream). Options to these APIs will remain stable within a major release of Delta Lake (e.g., 1.x.x).
See the online documentation for the releases and their compatibility with Apache Spark versions.

Data Storage Compatibility

Delta Lake guarantees backward compatibility for all Delta Lake tables (i.e., newer versions of Delta Lake will always be able to read tables written by older versions of Delta Lake). However, we reserve the right to break forward compatibility as new features are introduced to the transaction protocol (i.e., an older version of Delta Lake may not be able to read a table produced by a newer version).

Breaking changes in the protocol are indicated by incrementing the minimum reader/writer version in the Protocol action.

Roadmap

For the high-level Delta Lake roadmap, see Delta Lake 2022H1 roadmap.
For the detailed timeline, see the project roadmap.

Transaction Protocol

Delta Transaction Log Protocol document provides a specification of the transaction protocol.

Requirements for Underlying Storage Systems

Delta Lake ACID guarantees are predicated on the atomicity and durability guarantees of the storage system. Specifically, we require the storage system to provide the following.

Atomic visibility: There must be a way for a file to be visible in its entirety or not visible at all.
Mutual exclusion: Only one writer must be able to create (or rename) a file at the final destination.
Consistent listing: Once a file has been written in a directory, all future listings for that directory must return that file.

See the online documentation on Storage Configuration for details.

Concurrency Control

Delta Lake ensures serializability for concurrent reads and writes. Please see Delta Lake Concurrency Control for more details.

Reporting issues

We use GitHub Issues to track community reported issues. You can also contact the community for getting answers.

Contributing

We welcome contributions to Delta Lake. See our CONTRIBUTING.md for more details.

We also adhere to the Delta Lake Code of Conduct.

Building

Delta Lake is compiled using SBT. Ensure that your Java version is at least 17 (you can verify with java -version).

To compile, run

build/sbt compile

To generate artifacts, run

build/sbt package

To execute tests, run

build/sbt test

To execute a single test suite, run

build/sbt spark/'testOnly org.apache.spark.sql.delta.optimize.OptimizeCompactionSQLSuite'

To execute a single test within and a single test suite, run

build/sbt spark/'testOnly *.OptimizeCompactionSQLSuite -- -z "optimize command: on partitioned table - all partitions"'

Refer to SBT docs for more commands.

Running python tests locally

Setup Environment

Install Conda (Skip if you already installed it)

Follow Conda Download to install Anaconda.

Create an environment from environment file

Follow Create Environment From Environment file to create a Conda environment from <repo-root>/python/environment.yml and activate the newly created delta_python_tests environment.

# Note the `--file` argument should be a fully qualified path. Using `~` in file
# path doesn't work. Example valid path: `/Users/macuser/delta/python/environment.yml`

conda env create --name delta_python_tests --file=<absolute_path_to_delta_repo>/python/environment.yml`

JDK Setup

Build needs JDK 11. Make sure to setup JAVA_HOME that points to JDK 11.

Running tests

conda activate delta_python_tests
python3 <delta-root>/python/run-tests.py

IntelliJ Setup

IntelliJ is the recommended IDE to use when developing Delta Lake. To import Delta Lake as a new project:

Clone Delta Lake into, for example, ~/delta.
In IntelliJ, select File > New Project > Project from Existing Sources... and select ~/delta.
Under Import project from external model select sbt. Click Next.
Under Project JDK specify a valid Java 11 JDK and opt to use SBT shell for project reload and builds.
Click Finish.
In your terminal, run build/sbt clean package. Make sure you use Java 11. The build will generate files that are necessary for Intellij to index the repository.

Setup Verification

After waiting for IntelliJ to index, verify your setup by running a test suite in IntelliJ.

Search for and open DeltaLogSuite
Next to the class declaration, right click on the two green arrows and select Run 'DeltaLogSuite'

Troubleshooting

If you see errors of the form

Error:(46, 28) object DeltaSqlBaseParser is not a member of package io.delta.sql.parser
import io.delta.sql.parser.DeltaSqlBaseParser._
...
Error:(91, 22) not found: type DeltaSqlBaseParser
    val parser = new DeltaSqlBaseParser(tokenStream)

then follow these steps:

Ensure you are using Java 11. You can set this using

export JAVA_HOME=`/usr/libexec/java_home -v 11`

Compile using the SBT CLI: build/sbt clean compile.
Go to File > Project Structure... > Modules > delta-spark.
In the right panel under Source Folders remove any target folders, e.g. target/scala-2.12/src_managed/main [generated]
Click Apply and then re-run your test.

License

Apache License 2.0, see LICENSE.

Community

There are two mediums of communication within the Delta Lake community.

Public Slack Channel
- Register here
- Login here
Linkedin page
Youtube channel
Public Mailing list

sbt/sbt

sbt, the interactive build tool

https://scala-sbt.org

sbt

sbt is a build tool for Scala, Java, and more.

For general documentation, see https://www.scala-sbt.org/.

sbt 2.x

This is the 2.x series of sbt. The source code of sbt is split across several GitHub repositories, including this one.

sbt/io hosts sbt.io module.
sbt/zinc hosts Zinc, an incremental compiler for Scala.
sbt/sbt, this repository hosts modules that implement the build tool.

Issues and Pull Requests

Please read CONTRIBUTING carefully before opening a GitHub Issue or a pull request.

If you're looking for an idea for a contribution, issues labeled with good first issue or help wanted might be good starting points.

If you would like to ask questions about sbt, there's sbt channel on Scala Discord, but it would be good to gather questions on Stackoverflow.

license

See LICENSE.

NVIDIA/cudf-spark

NVIDIA cuDF for Apache Spark plugin - accelerate Apache Spark with GPUs

https://nvidia.github.io/cudf-spark

NVIDIA cuDF plugin for Apache Spark

NOTE: For the latest stable README.md ensure you are on the main branch.

The NVIDIA cuDF plugin for Apache Spark provides a plugin library that leverages GPUs to accelerate processing via the cuDF (CUDA DataFrame) libraries.

Documentation on the current release can be found here.

To get started and try the plugin out use the getting started guide.

Note: The NVIDIA cuDF plugin for Apache Spark was formerly known as the RAPIDS Accelerator for Apache Spark. The RAPIDS name will be sunset over time. Github links from spark-rapids will redirect to cudf-spark. Artifact names will remain the same for now.

Compatibility

The SQL plugin tries to produce results that are bit for bit identical with Apache Spark. Operator compatibility is documented here

Tuning

To get started tuning your job and get the most performance out of it please start with the tuning guide.

Configuration

The plugin has a set of Spark configs that control its behavior and are documented here.

Issues & Questions

We use github to track bugs, feature requests, and answer questions. File an issue for a bug or feature request. Ask or answer a question on the discussion board.

Download

The jar files for the most recent release can be retrieved from the download page.

Building From Source

See the build instructions in the contributing guide.

Testing

Tests are described here.

Integration

The cuDF plugin provides some APIs for doing zero copy data transfer into other GPU enabled applications. It is described here.

Currently, we are working with XGBoost to try to provide this integration out of the box.

You may need to disable RMM caching when exporting data to an ML library as that library will likely want to use all of the GPU's memory and if it is not aware of RMM it will not have access to any of the memory that RMM is holding.

Qualification and Profiling tools

The Qualification and Profiling tools have been moved to nvidia/spark-rapids-tools repo.

Please refer to Qualification tool documentation and Profiling tool documentation for more details on how to use the tools.

Dependency for External Projects

If you develop functionality on top of the cuDF plugin (we currently limit support to GPU-accelerated UDFs) we recommend you declare the distribution artifact as a provided dependency.

<dependency>
    <groupId>com.nvidia</groupId>
    <artifactId>rapids-4-spark_2.12</artifactId>
    <version>26.10.0-SNAPSHOT</version>
    <scope>provided</scope>
</dependency>

guardian/frontend

The Guardian DotCom.

https://theguardian.com

Frontend

The backend of the Guardian website frontend.

For everybody who engages with our journalism, theguardian.com is an industry-best news website that is fast, accessible and easy to use. Unlike other ways of developing products, ours puts the audience first.

Frontend is a set of Play Framework 2 Scala applications. It is built in two parts, using make for the client side asset build and SBT for the Play Framework backend.

Frontend's responsibility is to build JSON to pass to the modern rendering service, Dotcom Rendering.

Core Development Principles (lines in the sand)

These principles apply to all requests on www.theguardian.com and api.nextgen.guardianapps.co.uk (our Ajax URL)

Every request can be cached and has an appropriate Cache-Control header set.
Each request may only perform one I/O operation on the backend. (you cannot make two calls to the content API or any other 3rd party)
The average response time of any endpoint is less than 500ms.
Requests that take longer than two seconds will be terminated.

Documentation

All documentation notes and useful items can be found in the docs folder.

New developers

Welcome! How to set up and run frontend.

To get set up, please follow the installation guide.

Fixes for common problems can be found here.

Please read the development tips document to learn about more about development process.

For our deployment process, see how to deploy.

apache/kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

https://kyuubi.apache.org/

Project - Documentation - Who's using

Apache Kyuubi

Apache Kyuubi™ is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

What is Kyuubi?

Kyuubi provides a pure SQL gateway through Thrift JDBC/ODBC interface for end-users to manipulate large-scale data with pre-programmed and extensible Spark SQL engines. This "out-of-the-box" model minimizes the barriers and costs for end-users to use Spark at the client side. At the server-side, Kyuubi server and engines' multi-tenant architecture provides the administrators a way to achieve computing resource isolation, data security, high availability, high client concurrency, etc.

A HiveServer2-like API
Multi-tenant Spark Support
Running Spark in a serverless way

Target Users

Kyuubi's goal is to make it easy and efficient for anyone to use Spark(maybe other engines soon) and facilitate users to handle big data like ordinary data. Here, anyone means that users do not need to have a Spark technical background but a human language, SQL only. Sometimes, SQL skills are unnecessary when integrating Kyuubi with Apache Superset, which supports rich visualizations and dashboards.

In typical big data production environments with Kyuubi, there should be system administrators and end-users.

System administrators: A small group consists of Spark experts responsible for Kyuubi deployment, configuration, and tuning.
End-users: Focus on business data of their own, not where it stores, how it computes.

Additionally, the Kyuubi community will continuously optimize the whole system with various features, such as History-Based Optimizer, Auto-tuning, Materialized View, SQL Dialects, Functions, etc.

Usage scenarios

Port workloads from HiveServer2 to Spark SQL

In typical big data production environments, especially secured ones, all bundled services manage access control lists to restricting access to authorized users. For example, Hadoop YARN divides compute resources into queues. With Queue ACLs, it can identify and control which users/groups can take actions on particular queues. Similarly, HDFS ACLs control access of HDFS files by providing a way to set different permissions for specific users/groups.

Apache Spark is a unified analytics engine for large-scale data processing. It provides a Distributed SQL Engine, a.k.a, the Spark Thrift Server(STS), designed to be seamlessly compatible with HiveServer2 and get even better performance.

HiveServer2 can identify and authenticate a caller, and then if the caller also has permissions for the YARN queue and HDFS files, it succeeds. Otherwise, it fails. However, on the one hand, STS is a single Spark application. The user and queue to which STS belongs are uniquely determined at startup. Consequently, STS cannot leverage cluster managers such as YARN and Kubernetes for resource isolation and sharing or control the access for callers by the single user inside the whole system. On the other hand, the Thrift Server is coupled in the Spark driver's JVM process. This coupled architecture puts a high risk on server stability and makes it unable to handle high client concurrency or apply high availability such as load balancing as it is stateful.

Kyuubi extends the use of STS in a multi-tenant model based on a unified interface and relies on the concept of multi-tenancy to interact with cluster managers to finally gain the ability of resources sharing/isolation and data security. The loosely coupled architecture of the Kyuubi server and engine dramatically improves the client concurrency and service stability of the service itself.

DataLake/Lakehouse Support

The vision of Kyuubi is to unify the portal and become an easy-to-use data lake management platform. Different kinds of workloads, such as ETL processing and BI analytics, can be supported by one platform, using one copy of data, with one SQL interface.

Logical View support via Kyuubi DataLake Metadata APIs
Multiple Catalogs support
SQL Standard Authorization support for DataLake(coming)

Cloud Native Support

Kyuubi can deploy its engines on different kinds of Cluster Managers, such as, Hadoop YARN, Kubernetes, etc.

The Kyuubi Ecosystem(present and future)

The figure below shows our vision for the Kyuubi Ecosystem. Some of them have been realized, some in development, and others would not be possible without your help.

Online Documentation

Quick Start

Ready? Getting Started with Kyuubi.

Contributing

Project & Community Status

Aside

The project took its name from a character of a popular Japanese manga - Naruto. The character is named Kyuubi Kitsune/Kurama, which is a nine-tailed fox in mythology. Kyuubi spread the power and spirit of fire, which is used here to represent the powerful Apache Spark. Its nine tails stand for end-to-end multi-tenancy support of this project.

akka/akka-core

A platform to build and run apps that are elastic, agile, and resilient. SDK, libraries, and hosted environments.

https://doc.akka.io/libraries/

Akka

Akka is a powerful platform that simplifies building and operating highly responsive, resilient, and scalable services.

The platform consists of

the Akka SDK for straightforward, rapid development with AI assist and automatic clustering. Services built with the Akka SDK are automatically clustered and can be deployed on any infrastructure.
and Akka Automated Operations, a managed solution that handles everything for Akka SDK services from auto-elasticity to multi-region high availability running safely within your VPC.

The Akka SDK and Akka Automated Operations are built upon the foundational Akka libraries, providing the building blocks for distributed systems.

Akka has been downloaded more than 1 billion times and has powered thousands of systems for over 15 years. Akka enables millions of concurrent users, terabyte stream processing, low-latency read/write data access, 99.9999% availability, and multi-region high availability and disaster recovery (HA/DR).

Akka is used to build agentic AI, AI inference, transactional, analytical, digital twin, IOT, and edge-to-cloud systems. It is simple to learn and operate. Akka creates production-ready systems without requiring prior knowledge of events, threading, or distributed systems.

Akka core library

The Akka core library provides:

Multi-threaded behavior without the use of low-level concurrency constructs like atomics or locks — relieving you from even thinking about memory visibility issues.
Transparent remote communication between systems and their components — relieving you from writing and maintaining difficult networking code.
A clustered, high-availability architecture that is elastic, scales in or out, on demand — enabling you to deliver a truly reactive system.

At Akka's core is the actor model which provides a level of abstraction that makes it easier to write correct concurrent, parallel and distributed systems. The actor model spans the full set of Akka libraries, providing you with a consistent way of understanding and using them. Thus, Akka offers a depth of integration that you cannot achieve by picking libraries to solve individual problems and trying to piece them together.

Using the Actor Model we raise the abstraction level and provide a better platform to build correct concurrent and scalable applications. This model is a perfect match for the principles laid out in the Reactive Manifesto.

For resilience, we adopt the "Let it crash" model which the telecom industry has used with great success to build applications that self-heal and systems that never stop.

Akka actors also provide the abstraction for transparent distribution and the basis for truly scalable and fault-tolerant applications.

Reference Documentation

The current versions of all Akka libraries are listed on the Akka Dependencies page. Releases of the Akka core libraries in this repository are listed on the GitHub releases page.

The reference documentation for all Akka libraries is available via doc.akka.io/libraries/, details for the Akka core libraries for Scala and Java.

The current versions of all Akka libraries are listed on the Akka Dependencies page. Releases of the Akka core libraries in this repository are listed on the GitHub releases page.

Contributing

Contributions are very welcome!

If you see an issue that you'd like to see fixed, or want to shape out some ideas, the best way to make it happen is to help out by submitting a pull request implementing it. We welcome contributions from all, even you are not yet familiar with this project, We are happy to get you started, and will guide you through the process once you've submitted your PR.

Refer to the CONTRIBUTING.md file for more details about the workflow, and general hints on how to prepare your pull request. You can also ask for clarifications or guidance in GitHub issues directly, or in the akka/dev chat if a more real time communication would be of benefit.

License

Akka is licensed under the Business Source License 1.1, please see the Akka License FAQ.

Tests and documentation are under a separate license, see the LICENSE file in each documentation and test root directory for details.

playframework/playframework

The Community Maintained High Velocity Web Framework For Java and Scala.

http://www.playframework.com

Play Framework - The High Velocity Web Framework

The Play Framework combines productivity and performance making it easy to build scalable web applications with Java and Scala. Play is developer friendly with a "just hit refresh" workflow and built-in testing support. With Play, applications scale predictably due to a stateless and non-blocking architecture. By being RESTful by default, including assets compilers, JSON & WebSocket support, Play is a perfect fit for modern web & mobile applications.

Learn More

Sponsors & Backers

If you find Play useful for work, please consider asking your company to support this Open Source project by becoming a sponsor.
You can also individually sponsor the project by becoming a backer.

Thank you to our premium sponsors!

Thank you to all our backers!

Thank you for providing free tooling!

License

Copyright (C) from 2022 The Play Framework Contributors https://github.com/playframework, 2011-2021 Lightbend Inc. https://www.lightbend.com

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this project except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

scalapb/ScalaPB

Protocol buffer compiler for Scala.

https://scalapb.github.io/

ScalaPB

ScalaPB is a protocol buffer compiler (protoc) plugin for Scala. It will generate Scala case classes, parsers and serializers for your protocol buffers.

ScalaPB generates case classes that can co-exist in the same project alongside the Java-generated code for ProtocolBuffer. This makes it easy to gradually migrate an existing project from the Java version of protocol buffers to Scala. This is achieved by having the ScalaPB generated code use the proto file as part of the package name (in contrast to Java which uses the file name in CamelCase as an outer class)

Each top-level message and enum is written to a separate Scala file. This results in a significant improvement in incremental compilations.

Another cool feature of ScalaPB is that it can optionally generate methods that convert a Java protocol buffer to a Scala protocol buffer and vice versa. This is useful if you are gradually migrating a large code base from Java protocol buffers to Scala. The optional Java conversion is required if you want to use fromAscii (parsing ASCII representation of a protocol buffer). The current implementation delegates to the Java version.

Highlights

Supports proto2 and proto3
Easily update nested structure in functional way using lenses
Scala.js integration
GRPC integration
Compatible with SparkSQL (through a helper library)
Conversion to and from JSON
Support user-defined options (since 0.5.29)
Support extensions (since 0.6.0)

Versions

Version	Description
0.6.x	Stable. Supports Protobuf 2.6.x to 3.5.x
0.5.x	Supports Protobuf 2.6.x and Protobuf 3.1.x.
0.4.x	Stable, unsupported. Works with Protobuf 2.6.x

Installing

To automatically generate Scala case classes for your messages add ScalaPB's sbt plugin to your project. Create a file named project/protoc.sbt containing the following line:

addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.6")

libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.11.3"

Add the following line to your build.sbt:

Compile / PB.targets := Seq(
  scalapb.gen() -> (Compile / sourceManaged).value / "scalapb"
)

For additional configuration options, see ScalaPB SBT Settings documentation

Using ScalaPB

Documentation is available at ScalaPB website.

Questions?

See contacting us.

Testing

ScalaPB uses ScalaCheck to aggressively test the generated code. The test generates many different sets of proto files. The sets are growing in complexity: number of files, references to messages from other protos, message nesting and so on. Then, test data is generated to populate this protocol schema, then we check that the ScalaPB generated code behaves exactly like the reference implementation in Java.

Running the tests:

sbt test

The tests take a few minutes to run. There is a smaller test suite called e2e that uses the sbt plugin to compile the protos and runs a series of ScalaChecks on the outputs. To run it:

./e2e.sh

gatling/gatling

Modern Load Testing as Code

https://gatling.io

Gatling

Load Testing & Performance Engineering Platform

Gatling is an open-source load testing platform designed for modern performance engineering.

It helps teams simulate real-world traffic, validate system behavior under load, detect regressions early, and make confident release decisions.

Gatling officially supports HTTP, WebSocket, Server-Sent Events, JMS, gRPC, and MQTT, and integrates natively with CI/CD pipelines, APM tools, and modern DevOps workflows.

What is Gatling?

Gatling started as a high-performance, code-first load testing tool.

It has evolved into the foundation of a broader vision: Continuous Performance Intelligence, helping teams move from simply running tests to structuring performance around intent, measurable objectives, and decision-making.

Performance is no longer a technical checkbox. It’s a business imperative that drives revenue, ensures reliability, and manages risk.

Gatling enables you to:

Define performance tests as code (write tests in Java, JavaScript, TypeScript, Kotlin, or Scala, using fluent and expressive SDKs)
Compare runs and detect regressions
Monitor SLOs and performance objectives
Integrate with observability tools
Automate performance gates in CI/CD

Why Gatling?

Many legacy tools rely on blocking IO and one-thread-per-user architectures, forcing teams to maintain large farms of injection servers.

Gatling uses a non-blocking, asynchronous architecture that maximizes resource efficiency and scalability per node.

If you’re:

Tired of maintaining heavy infrastructure just to generate load
Frustrated with complex GUIs and proprietary test formats
Looking for an expressive SDK instead of XML dumps
Wanting performance tests that live in your version control system

Gatling is built for you.

Getting Started

The fastest way to get started is to follow the official Gatling Tutorials

If you prefer jumping directly into SDK installation:

Java SDK

Java JVM Installation Guide

JavaScript/TypeScript SDK

JavaScript Installation Guide

Core Capabilities

Gatling provides the foundation for structured performance engineering.

1. Smart Performance Analysis

With Gatling Enterprise Edition, teams gain real-time dashboards, trend analysis, SLO monitoring, AI-assisted summaries, and integrations with leading APM tools.

2. Flexible Test Creation

Use JavaScript/TypeScript, Java, Scala, or Kotlin. Record browser sessions, build visually, and export to code. Author tests in your IDE with AI-assisted tooling.

Tests remain version-controlled, reviewable, and CI/CD friendly.

3. Continuous Load Testing

Integrate performance testing into build pipelines and release workflows.

Trigger simulations automatically, define stop criteria, and enforce performance gates before deployment. Manage configurations and infrastructure as code.

4. Adaptive Load Infrastructure

Run load wherever you need it: locally, in your cloud, on-premise, or via managed infrastructure. Scale globally while keeping traffic private and under cost control.

5. Collaboration & Governance

Gatling Enterprise Edition extends the open-source foundation with:

Role-based access control
SSO
Quotas and usage management
Shared reports and public links
Slack, Teams, and Jira integrations
Low-code or no-code test creation

One platform for Dev, QA, Ops, and leadership.

Integrations

Gatling integrates with leading CI/CD systems, APM tools, and observability platforms.

Discover all supported integrations

Community & Support

Found a bug? Raise an issue.

License

Gatling Open Source is licensed under Apache 2.0.

Partners

zio/zio

ZIO — A type-safe, composable library for async and concurrent programming in Scala

https://zio.dev

Project Stage	CI	Release	Snapshot	Issues

Scaladoc	Scaladex	Discord	Twitter	Gitpod	Gurubase
Scaladoc

Welcome to ZIO

ZIO is a zero-dependency Scala library for asynchronous and concurrent programming.

Powered by highly-scalable, non-blocking fibers that never waste or leak resources, ZIO lets you build scalable, resilient, and reactive applications that meet the needs of your business.

High-performance. Build scalable applications with minimal runtime overhead.
Type-safe. Use the full power of the Scala compiler to catch bugs at compile time.
Concurrent. Easily build concurrent apps without deadlocks, race conditions, or complexity.
Asynchronous. Write sequential code that looks the same whether it's asynchronous or synchronous.
Resource-safe. Build apps that never leak resources (including threads!), even when they fail.
Testable. Inject test services into your app for fast, deterministic, and type-safe testing.
Resilient. Build apps that never lose errors, and which respond to failure locally and flexibly.
Functional. Rapidly compose solutions to complex problems from simple building blocks.

To learn more about ZIO, see the following references:

Adopters

Following is a partial list of companies happily using ZIO in production to craft concurrent applications.

Want to see your company here? Submit a PR!

AdGear / Samsung Ads
Adidas
adpulse.io
adsquare
Amazon
Anduin Transactions
AnyMind Group
Ayolab
Asana
Aurinko
auto.ru
AutoScout24
Avast
Bank of America
Believe
Blue Insight Digital
BlueLabs
Bpp
Broad
Caesars Digital
CalcBank
Call Handling
Carvana
Cellular
ChannelPilot
Chartboost
Cloudfarms
CodeComprehension
Codept
Colisweb
Collibra
Compellon
Complicated Robot
Conduktor
Contramap
Coralogix
Credit Karma
CurrencyCloud
D.E.Solution
DaSCH
DataChef
Demandbase
Demyst
Devsisters
DHL Parcel The Netherlands
Disney+ Streaming
Doomoolmori
Dow Jones
DPG recruitment
Dream11
Deutsche Telekom IoT GmbH
eBay
Eaglescience
Electricité de France (EDF)
EnelX
Evolution
Evo
Flipp
Fugo.ai
Garner Distributed Workflow
Glean
GrandParade
greyflower.media GmbH
Hunters.AI
Hypefactors
iHeartRadio
IHS Markit
Investsuite
Invia
Kaizen Solutions
Kamon APM
Kodmagi
Kensu
LambdaWorks
LeadIQ
Lernkunst
LiveIntent Inc.
Lottoland
MATECHS
Megogo
MOIA
Mylivn
NewMotion
Nexxchange
Nike
NsLookup
Ocado Technology
Olyro GmbH
Optrak
Performance Immo
Pirum
PITS Global Data Recovery Services
Playtika
PPC Samurai
Prezi
Radix Labs
Railroad19
Randstad Groep Nederland
Rapidor
PIM Solutions
Remember The Milk
REWE Digital
LKWfee
Risk Ident
Rocker
Rudder
Samsung Food
Sanjagh
Scalac
SecurityScorecard
SentinelOne
Signicat
Société Générale Corporate and Investment Banking
SoftwareMill
StreamWeaver
Stuart
Teads
The Pokemon Company International
TomTom
Tinka
Tinkoff
Trackabus
Trainor
Tranzzo
TreuTech
Tweddle Group
Undo
Unit
Univalence
Unzer
Vakantiediscounter
Verbund AG
Waylay
Wehkamp
Wefunder
Wolt
Yandex.Classifieds
Audela
Valamis Group
Valsea
VirtusLab
Vish
Vivid Money
Zalando
Zooz

Learn More on the ZIO Homepage

Code of Conduct

See the Code of Conduct

Support

Come chat with us on .

Legal

gitbucket/gitbucket

A Git platform powered by Scala with easy installation, high extensibility & GitHub API compatibility

https://gitbucket.github.io/

GitBucket

GitBucket is a Git web platform powered by Scala offering:

Easy installation
Intuitive UI
High extensibility by plugins
API compatibility with GitHub

Features

The current version of GitBucket provides many features such as:

Public / Private Git repositories (with http/https and ssh access)
GitLFS support
Repository viewer including an online file editor
Issues, Pull Requests and Wiki for repositories
Activity timeline and email notifications
Account and group management with LDAP integration
a Plug-in system

Installation

GitBucket requires Java 17. You have to install it, if it is not already installed.

Download the latest gitbucket.war from the releases page and run it by java -jar gitbucket.war.
Go to http://[hostname]:8080/ and log in with ID: root / Pass: root.

You can also deploy gitbucket.war to a servlet container which supports Servlet 3.0 (like Jetty, Tomcat, JBoss, etc). Note that GitBucket doesn't support Jakarta EE yet.

For more information about installation on Mac or Windows Server (with IIS), or configuration of Apache or Nginx and also integration with other tools or services such as Jenkins or Slack, see Wiki.

To upgrade GitBucket, replace gitbucket.war with the new version, after stopping GitBucket. All GitBucket data is stored in HOME/.gitbucket by default. So if you want to back up GitBucket's data, copy this directory to the backup location.

Plugins

GitBucket has a plug-in system that allows extra functionality. Officially the following plug-ins are provided:

You can find more plugins made by the community at GitBucket community plugins.

Building and Development

If you want to try the development version of GitBucket, or want to contribute to the project, please see the Developer's Guide. It provides instructions on building from source and on setting up an IDE for debugging. It also contains documentation of the core concepts used within the project.

Support

If you have any questions about GitBucket, see Wiki and check issues whether there is a same question or request in the past.
If you can't find same question and report, send it to our Gitter chat room before raising an issue.
The highest priority of GitBucket is the ease of installation and API compatibility with GitHub, so your feature request might be rejected if they go against those principles.

What's New in 4.47.x

4.47.0 - 9 Aug 2026

Add more security options for creating repositories by cloning
Make the reset password token expiration period configurable
Improve GitHub API compatibility for repository IDs, fork management
Fix issue assignees not being preserved when a repository is renamed or transferred
Improvements and bug fixes for markdown format actions in the text editor

Note 1: This release includes a large database migration affecting core tables such as REPOSITORY and ACCOUNT. We strongly recommend backing up your database before upgrading to GitBucket 4.47.0.

Note 2: You have to migrate h2 database file if you will upgrade GitBucket from 4.42 or before to 4.43 or later and you are using the default h2 database because h2 1.x and h2.x don't have compatibility: https://www.h2database.com/html/migration-to-v2.html

It can't be done automatically using GitBucket's auto migration mechanism because it relies on database itself. So, users who use h2 will have to dump and recreate their database manually with the following steps:

# Export database using the current version of H2
$ curl -O https://repo1.maven.org/maven2/com/h2database/h2/1.4.199/h2-1.4.199.jar
$ java -cp h2-1.4.199.jar org.h2.tools.Script -url "jdbc:h2:~/.gitbucket/data" -user sa -password sa -script dump.sql

# Recreate database using the new version of H2
$ curl -O https://repo1.maven.org/maven2/com/h2database/h2/2.3.232/h2-2.3.232.jar
$ java -cp h2-2.3.232.jar org.h2.tools.RunScript -url "jdbc:h2:~/.gitbucket/data" -user sa -password sa -script dump.sql

In addition, if ~/.gitbucket/database.conf has the following configuration, remove ;MVCC=true from url.

db {
  url = "jdbc:h2:${DatabaseHome};MVCC=true" // => "jdbc:h2:${DatabaseHome}"
  ...
}

See the change log for all the past updates.

apache/pekko

Build highly concurrent, distributed, and resilient message-driven applications using Java/Scala

https://pekko.apache.org/

Apache Pekko

Apache Pekko is an open-source framework for building applications that are concurrent, distributed, resilient and elastic. Pekko uses the Actor Model to provide more intuitive high-level abstractions for concurrency. Using these abstractions, Pekko also provides libraries for persistence, streams, HTTP, and more.

Pekko is a fork of Akka 2.6.x, prior to the Akka project's adoption of the Business Source License. Pekko now contains some features from Akka 2.7.0 (Apache 2.0 Licensed now)

Reference Documentation

See https://pekko.apache.org for the documentation including the API docs. The docs for all the Apache Pekko modules can be found there.

Building from Source

The CI build is Linux based (Ubuntu) and most Pekko developers use Macs or Linux machines. There have been reports of issues when building with Windows (#829).

Using IntelliJ IDEA

To use IntelliJ IDEA, you need to configure your project to use JDK 17. For a visual guide, refer to this Q&A.

File > Project Structure > Project Settings > Project > Set your JDK to version 17
Settings > ... > Scala Compiler Server > Set your JDK to version 17
Settings > ... > sbt > Set your JRE to version 17

Prerequisites

Make sure you have installed a Java Development Kit (JDK) version 17 or later.
Make sure you have sbt installed and using this JDK.
Graphviz is needed for the scaladoc generation build task, which is part of the release.

Running the Build

Open a command window and change directory to your preferred base directory
Use git to clone the repo or download a source release from https://pekko.apache.org (and unzip or untar it, as appropriate)
Change directory to the directory where you installed the source (you should have a file called build.sbt in this directory)
sbt compile compiles the main source for project default version of Scala (2.13)
- sbt +compile will compile for all supported versions of Scala
sbt test will compile the code and run the unit tests
sbt testQuick similar to test but when repeated in shell mode will only run failing tests
sbt testQuickUntilPassed similar to testQuick but will loop until tests pass.
sbt package will build the jars
- the jars will be built into target dirs of the various modules
- for the 'actor' module, the jar will be built to actor/target/scala-2.13/
sbt publishLocal will push the jars to your local Apache Ivy repository
sbt publishM2 will push the jars to your local Apache Maven repository
sbt docs/paradox will build the docs (the ones describing the module features)
- sbt docs/paradoxBrowse does the same but will open the docs in your browser when complete
- the index.html file will appear in target/paradox/site/main/
sbt unidoc will build the Javadocs for all the modules and load them to one place (may require Graphviz, see Prerequisites above)
- the index.html file will appear in target/scala-2.13/unidoc/
sbt sourceDistGenerate will generate source release to target/dist/
The version number that appears in filenames and docs is derived, by default. The derived version contains the most git commit id or the date/time (if the directory is not under git control).
- You can set the version number explicitly when running sbt commands
  - eg sbt "set ThisBuild / version := \"1.0.0\"; sourceDistGenerate"
- Or you can add a file called version.sbt to the same directory that has the build.sbt containing something like
  - ThisBuild / version := "1.0.0"

Community

There are several ways to interact with the Pekko community:

GitHub discussions: for questions and general discussion.
Pekko dev mailing list: for Pekko development discussions.
Pekko users mailing list: for Pekko user discussions.
GitHub issues: for bug reports and feature requests. Please search the existing issues before creating new ones. If you are unsure whether you have found a bug, consider asking in GitHub discussions or the mailing list first.

Contributing

Contributions are very welcome. If you have an idea on how to improve Pekko, don't hesitate to create an issue or submit a pull request.

See CONTRIBUTING.md for details on the development workflow and how to create your pull request.

Code of Conduct

Apache Pekko is governed by the Apache code of conduct. By participating in this project you agree to abide by its terms.

License

Apache Pekko is available under the Apache License, version 2.0. See LICENSE file for details.

apache/datafusion-comet

Apache DataFusion Comet Spark Accelerator

https://datafusion.apache.org/comet

Apache DataFusion Comet

Apache DataFusion Comet is a high-performance accelerator for Apache Spark. Comet keeps Spark queries Arrow-native end-to-end: operators, expressions, shuffle, and broadcast all stay in Apache Arrow columnar format, avoiding the per-row overhead of Spark's row-based engine. Within the Arrow-native pipeline, operators and expressions execute as Rust code (via the Apache DataFusion query engine) or as JVM code that operates directly on Arrow batches. Comet integrates with the Spark ecosystem without requiring any code changes.

Comet provides a ~2x speedup for TPC-DS @ SF 1000 (1TB), resulting in ~50% cost savings.

That 2x speedup gives you a choice: finish the same Spark workload in half the time on the cluster you already have, or match your current Spark performance on roughly half the resources. Either way, the gain translates directly into lower cloud bills, reduced on-prem capacity, and lower energy usage, with no changes to your existing Spark SQL, DataFrame, or PySpark code. Comet runs on commodity hardware: no GPUs, FPGAs, or other specialized accelerators are required, so the savings come from better utilization of the infrastructure you already run on.

See the Comet Benchmarking Guide for more details.

What Comet Accelerates

Comet accelerates Spark workloads by replacing Spark operators and expressions with high-performance implementations that process Apache Arrow columnar data directly. Most operators are powered by native Rust execution built on Apache DataFusion, while others run efficiently in the JVM on Arrow batches. This unified columnar execution model keeps processing within the Comet engine end-to-end, reducing overhead and delivering faster, more efficient query execution without reverting to Spark's traditional row-based engine.

Parquet scans: native Parquet reader integrated with Spark's query planner
Apache Iceberg: accelerated Parquet scans when reading Iceberg tables from Spark (see the Iceberg guide)
Shuffle: Arrow-IPC columnar shuffle with support for hash and range partitioning, in a native Rust implementation paired with a JVM fallback for unsupported partition key types
Expressions: hundreds of supported Spark expressions across math, string, datetime, array, map, JSON, hash, and predicate categories
Aggregations: hash aggregate with support for FILTER (WHERE ...) clauses
Joins: hash join, sort-merge join, and broadcast join
Scala/Java UDFs: support for keeping Scala/Java scalar UDFs in the Comet pipeline via Spark's whole-stage codegen (see the Scala UDF guide)

For the authoritative lists, see the supported expressions and supported operators pages.

Drop-In Integration

Comet is designed as a drop-in accelerator for Apache Spark, allowing you to integrate Comet into your existing Spark deployments and workflows seamlessly. With no code changes required, you can immediately harness the benefits of Comet's acceleration capabilities without disrupting your Spark applications.

Getting Started

Comet supports Apache Spark 3.4, 3.5, 4.0, and 4.1, and provides experimental support for Spark 4.2. See the installation guide for the detailed version, Java, and Scala compatibility matrix.

Install Comet by adding the jar for your Spark and Scala version to the Spark classpath and enabling the plugin. A typical configuration looks like:

export COMET_JAR=/path/to/comet-spark-spark3.5_2.12-<version>.jar

$SPARK_HOME/bin/spark-shell \
    --jars $COMET_JAR \
    --conf spark.driver.extraClassPath=$COMET_JAR \
    --conf spark.executor.extraClassPath=$COMET_JAR \
    --conf spark.plugins=org.apache.spark.CometPlugin \
    --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
    --conf spark.comet.explain.fallback.enabled=true \
    --conf spark.memory.offHeap.enabled=true \
    --conf spark.memory.offHeap.size=4g

For full installation instructions, published jar downloads, and configuration reference, see the installation guide and the configuration reference.

Community

Join the DataFusion Slack and Discord channels to connect with other users, ask questions, and share your experiences with Comet.

Contributing

We welcome contributions from the community to help improve and enhance Apache DataFusion Comet. Whether it's fixing bugs, adding new features, writing documentation, or optimizing performance, your contributions are invaluable in shaping the future of Comet. Check out our contributor guide to get started.

License

Apache DataFusion Comet is licensed under the Apache License 2.0. See the LICENSE.txt file for details.

ValeLang/Vale

Compiler for the Vale programming language - http://vale.dev/

https://vale.dev/

Vale

Vale is a programming language that's aims to be not only fast and memory-safe, but also easy and flexible.

Fast: Vale is AOT compiled to LLVM, statically-typed, and uses the new generational references technique for memory safety with speed and flexibility, and will soon have region borrow checking to make it even faster.
Safe: It is the safest native language, thanks to generational references and Fearless FFI.
Easy: Vale has memory-safe single ownership without garbage collection or a borrow checker, which makes it easy to write safe, fast code.

See http://vale.dev/ for more information on the language, and see our treasure trove of working examples, grouped by functionality.

Building a Vale Program

Download the latest binary from https://github.com/ValeLang/Vale/releases
Unzip it into, for example, ~/Vale, and cd into it.
Make a file hello.vale containing some vale code, like exported func main() { println("Hello world!"); }
Compile a program: ./valec build mymod=hello.vale --output_dir target
Run the program: target/main, and you should see the output Hello world!

Learning Vale

See the Guide for how to use Vale.

Building the compiler itself

For instructions for building the compiler itself, see Building the Compiler.

For an overview of the project structure, see Compiler Overview.

Editor plugins

VSCode plugin: Syntax-highlighting and basic autocompletion.
Vim plugin: Syntax-highlighting

Thank you to all of our sponsors!

Notes

Vale started in January 2013, and back then we called it "VLang", though there's now another language with that name. We then called it GelLLVM, in honor of Gel, the first language to offer constraint references. Since then, we've settled on the name "Vale". Note that Vale and Vala are two different languages.

akka/akka-http

The Streaming-first HTTP server/module of Akka

https://doc.akka.io/libraries/akka-http/current/

Akka

Akka is a powerful platform that simplifies building and operating highly responsive, resilient, and scalable services.

The platform consists of

the Akka SDK for straightforward, rapid development with AI assist and automatic clustering. Services built with the Akka SDK are automatically clustered and can be deployed on any infrastructure.
and Akka Automated Operations, a managed solution that handles everything for Akka SDK services from auto-elasticity to multi-region high availability running safely within your VPC.

The Akka SDK and Akka Automated Operations are built upon the foundational Akka libraries, providing the building blocks for distributed systems.

Akka HTTP library

The Akka HTTP modules implement a full server- and client-side HTTP stack on top of akka-actor and akka-stream. It's not a web-framework but rather a more general toolkit for providing and consuming HTTP-based services. While interaction with a browser is of course also in scope it is not the primary focus of Akka HTTP.

Akka HTTP follows a rather open design and many times offers several different API levels for "doing the same thing". You get to pick the API level of abstraction that is most suitable for your application. This means that, if you have trouble achieving something using a high-level API, there's a good chance that you can get it done with a low-level API, which offers more flexibility but might require you to write more application code.

Reference Documentation

The reference documentation for all Akka libraries is available via doc.akka.io/libraries/, details for the Akka HTTP library for Scala and Java.

The current versions of all Akka libraries are listed on the Akka Dependencies page. Releases of the Akka HTTP library in this repository are listed on the GitHub releases page.

Contributing

Contributions are very welcome!

If you see an issue that you'd like to see fixed, the best way to make it happen is to help out by submitting a pull request. For ideas of where to contribute, tickets marked as "help wanted" are a good starting point.

Maintenance

This project is maintained by the core Akka Team as well as the extended Akka HTTP Team, consisting of excellent and experienced developers who have shown their dedication and knowledge about HTTP and the codebase. This team may grow dynamically, and it is possible to propose new members to it.

Joining the extended team in such form gives you, in addition to street-cred, of course committer rights to this repository as well as higher impact onto the roadmap of the project. Come and join us!

License

Akka is licensed under the Business Source License 1.1, please see the Akka License FAQ.

Tests and documentation are under a separate license, see the LICENSE file in each documentation and test root directory for details.

chipsalliance/chisel

Chisel: A Modern Hardware Design Language

https://www.chisel-lang.org/

The Constructing Hardware in a Scala Embedded Language (Chisel) is an open-source hardware description language (HDL) used to describe digital electronics and circuits at the register-transfer level that facilitates advanced circuit generation and design reuse for both ASIC and FPGA digital logic designs.

Chisel adds hardware construction primitives to the Scala programming language, providing designers with the power of a modern programming language to write complex, parameterizable circuit generators that produce synthesizable Verilog. This generator methodology enables the creation of re-usable components and libraries, such as the FIFO queue and arbiters in the Chisel Standard Library, raising the level of abstraction in design while retaining fine-grained control.

For more information on the benefits of Chisel see: "What benefits does Chisel offer over classic Hardware Description Languages?"

Chisel is powered by FIRRTL (Flexible Intermediate Representation for RTL), a hardware compiler framework implemented by LLVM CIRCT.

Chisel is permissively licensed (Apache 2.0) under the guidance of CHIPS Alliance.

What does Chisel code look like?

LED blink

import chisel3._
import chisel3.util.Counter
import circt.stage.ChiselStage

class Blinky(freq: Int, startOn: Boolean = false) extends Module {
  val io = IO(new Bundle {
    val led0 = Output(Bool())
  })
  // Blink LED every second using Chisel built-in util.Counter
  val led = RegInit(startOn.B)
  val (_, counterWrap) = Counter(true.B, freq / 2)
  when(counterWrap) {
    led := ~led
  }
  io.led0 := led
}

object Main extends App {
  // These lines generate the Verilog output
  println(
    ChiselStage.emitSystemVerilog(
      new Blinky(1000),
      firtoolOpts = Array("-disable-all-randomization", "-strip-debug-info")
    )
  )
}

Should output the following Verilog:

Click to expand!

// Generated by CIRCT firtool-1.37.0
module Blinky(
  input  clock,
         reset,
  output io_led0
);

  reg       led;
  reg [8:0] counterWrap_c_value;
  always @(posedge clock) begin
    if (reset) begin
      led <= 1'h0;
      counterWrap_c_value <= 9'h0;
    end
    else begin
      automatic logic counterWrap = counterWrap_c_value == 9'h1F3;
      led <= counterWrap ^ led;
      if (counterWrap)
        counterWrap_c_value <= 9'h0;
      else
        counterWrap_c_value <= counterWrap_c_value + 9'h1;
    end
  end // always @(posedge)
  assign io_led0 = led;
endmodule

FIR Filter

Consider an FIR filter that implements a convolution operation, as depicted in this block diagram:

While Chisel provides similar base primitives as synthesizable Verilog, and could be used as such:

// 3-point moving sum implemented in the style of a FIR filter
class MovingSum3(bitWidth: Int) extends Module {
  val io = IO(new Bundle {
    val in = Input(UInt(bitWidth.W))
    val out = Output(UInt(bitWidth.W))
  })

  val z1 = RegNext(io.in)
  val z2 = RegNext(z1)

  io.out := (io.in * 1.U) + (z1 * 1.U) + (z2 * 1.U)
}

the power of Chisel comes from the ability to create generators, such as an FIR filter that is defined by the list of coefficients:

// Generalized FIR filter parameterized by the convolution coefficients
class FirFilter(bitWidth: Int, coeffs: Seq[UInt]) extends Module {
  val io = IO(new Bundle {
    val in = Input(UInt(bitWidth.W))
    val out = Output(UInt(bitWidth.W))
  })
  // Create the serial-in, parallel-out shift register
  val zs = Reg(Vec(coeffs.length, UInt(bitWidth.W)))
  zs(0) := io.in
  for (i <- 1 until coeffs.length) {
    zs(i) := zs(i-1)
  }

  // Do the multiplies
  val products = VecInit.tabulate(coeffs.length)(i => zs(i) * coeffs(i))

  // Sum up the products
  io.out := products.reduce(_ + _)
}

and use and re-use them across designs:

val movingSum3Filter = Module(new FirFilter(8, Seq(1.U, 1.U, 1.U)))  // same 3-point moving sum filter as before
val delayFilter = Module(new FirFilter(8, Seq(0.U, 1.U)))  // 1-cycle delay as a FIR filter
val triangleFilter = Module(new FirFilter(8, Seq(1.U, 2.U, 3.U, 2.U, 1.U)))  // 5-point FIR filter with a triangle impulse response

The above can be converted to Verilog using ChiselStage:

import chisel3.stage.ChiselGeneratorAnnotation
import circt.stage.{ChiselStage, FirtoolOption}

(new ChiselStage).execute(
  Array("--target", "systemverilog"),
  Seq(ChiselGeneratorAnnotation(() => new FirFilter(8, Seq(1.U, 1.U, 1.U))),
    FirtoolOption("--disable-all-randomization"))
)

Alternatively, you may generate some Verilog directly for inspection:

val verilogString = chisel3.getVerilogString(new FirFilter(8, Seq(0.U, 1.U)))
println(verilogString)

Getting Started

Bootcamp Interactive Tutorial

The online Chisel Bootcamp is the recommended way to get started with and learn Chisel. No setup is required (it runs in the browser), nor does it assume any prior knowledge of Scala.

The classic Chisel tutorial contains small exercises and runs on your computer.

A Textbook on Chisel

If you like a textbook to learn Chisel and also a bit of digital design in general, you may be interested in reading Digital Design with Chisel. It is available in English, Chinese, Japanese, and Vietnamese.

Build Your Own Chisel Projects

Please see the Installation page of the Chisel website for information about how to use Chisel locally.

When you're ready to build your own circuits in Chisel, we recommend starting from the Chisel Template repository, which provides a pre-configured project, example design, and testbench. Follow the chisel-template README to get started.

If you insist on setting up your own project from scratch, your project needs to depend on both the chisel-plugin (Scalac plugin) and the chisel library. For example, in SBT this could be expressed as:

// build.sbt
scalaVersion := "2.13.16"
val chiselVersion = "7.2.0"
addCompilerPlugin("org.chipsalliance" % "chisel-plugin" % chiselVersion cross CrossVersion.full)
libraryDependencies += "org.chipsalliance" %% "chisel" % chiselVersion

For Chisel prior to v5.0.0, Chisel was published using a different artifact name:

// build.sbt
scalaVersion := "2.13.10"
addCompilerPlugin("edu.berkeley.cs" % "chisel3-plugin" % "3.6.0" cross CrossVersion.full)
libraryDependencies += "edu.berkeley.cs" %% "chisel3" % "3.6.0"
// We also recommend using chiseltest for writing unit tests
libraryDependencies += "edu.berkeley.cs" %% "chiseltest" % "0.6.0" % "test"

Guide For New Contributors

If you are trying to make a contribution to this project, please read CONTRIBUTING.md.

Design Verification

These simulation-based verification tools are available for Chisel:

svsim is the lightweight testing library for Chisel, included in this repository.
chiseltest (Chisel 6.0 and before) is the batteries-included testing and formal verification library for Chisel-based RTL designs and a replacement for the former PeekPokeTester, providing the same base constructs but with a streamlined interface and concurrency support with fork and join with internal and Verilator integration for simulations.

Documentation

Useful Resources

Cheat Sheet, a 2-page reference of the base Chisel syntax and libraries
ScalaDoc (latest), a listing, description, and examples of the functionality exposed by Chisel, older versions are also available
Gitter, where you can ask questions or discuss anything Chisel
Website (source)
Scastie (v6.0.0) - cannot generate Verilog (firtool does not work in Scastie)
Scastie (v3.6.0) - generates Verilog with legacy Scala FIRRTL Compiler
asic-world If you aren't familiar with verilog, this is a good tutorial.

If you are migrating from Chisel2, see the migration guide.

Chisel Dev Meeting

Chisel/FIRRTL development meetings happen every Monday from 9:00-10:00 am PT.

Call-in info and meeting notes are available here.

Data Types Overview

These are the base data types for defining circuit components:

Contributor Documentation

This section describes how to get started contributing to Chisel itself, including how to test your version locally against other projects that pull in Chisel using sbt's managed dependencies.

Useful Resources for Contributors

The Useful Resources for users are also helpful for contributors.

Chisel Breakdown Slides, an introductory talk about Chisel's internals

Compiling and Testing Chisel

You must first install required dependencies to build Chisel locally, please see the installation instructions.

Clone and build the Chisel library:

git clone https://github.com/chipsalliance/chisel.git
cd chisel
./mill chisel[].compile

In order to run the following unit tests, you will need several tools on your PATH, namely verilator, yosys, espresso, slang, and filecheck. Check that each is installed on your PATH by running which verilator and so on.

If the compilation succeeded and the dependencies noted above are installed, you can then run the included unit tests by invoking:

# Run the ScalaTest unit tests:
./mill chisel[].test
# Run the Chisel unit tests (modules marked with chisel3.UnitTest trait):
./mill chisel[].unitTest

Running Projects Against Local Chisel

To use the development version of Chisel (main branch), you will need to build from source and publish locally. The repository version can be found by running ./mill show unipublish[2.13].publishVersion. As of the time of writing it was: 7.1.1+16-767b9eb3-SNAPSHOT

To publish your version of Chisel to the local Ivy repository, run:

./mill unipublish[2.13].publishLocal

The compiled version gets placed in ~/.ivy2/local/org.chipsalliance/. If you need to un-publish your local copy of Chisel, remove the directory generated in ~/.ivy2/local/org.chipsalliance/.

In order to have your projects use this version of Chisel, you should update the libraryDependencies setting in your project's build.sbt file to use the current version, for example:

val chiselVersion = "7.1.1+16-767b9eb3-SNAPSHOT"
addCompilerPlugin("org.chipsalliance" % "chisel-plugin" % chiselVersion cross CrossVersion.full)
libraryDependencies += "org.chipsalliance" %% "chisel" % chiselVersion

Chisel Architecture Overview

The Chisel compiler consists of these main parts:

The frontend, chisel3.*, which is the publicly visible "API" of Chisel and what is used in Chisel RTL. These just add data to the...
The Builder, chisel3.internal.Builder, which maintains global state (like the currently open Module) and contains commands, generating...
The intermediate data structures, chisel3.firrtl.*, which are syntactically very similar to Firrtl. Once the entire circuit has been elaborated, the top-level object (a Circuit) is then passed to...
The Firrtl emitter, chisel3.firrtl.Emitter, which turns the intermediate data structures into a string that can be written out into a Firrtl file for further processing.

Also included is:

The standard library of circuit generators, chisel3.util.*. These contain commonly used interfaces and constructors (like Decoupled, which wraps a signal with a ready-valid pair) as well as fully parameterizable circuit generators (like arbiters and multiplexors).
Chisel Stage, chisel3.stage.*, which contains compilation and test functions that are invoked in the standard Verilog generation and simulation testing infrastructure. These can also be used as part of custom flows.

Chisel Sub-Projects

Chisel consists of several Scala projects; each is its own separate compilation unit:

core is the bulk of the source code of Chisel, depends on firrtl, svsim, and macros
firrtl is the vestigial remains of the old Scala FIRRTL compiler, much if it will likely be absorbed into core
macros is most of the macros used in Chisel, no internal dependencies
plugin is the compiler plugin, no internal dependencies
src/main is the "main" that brings it all together and includes a util library, which depends on core
svsim is a low-level library for compiling and controlling SystemVerilog simulations, currently targeting Verilator and VCS as backends

Code that touches lots of APIs that are private to the chisel3 package should belong in core, while code that is pure Chisel should belong in src/main.

Which version should I use?

We encourage Chisel users (as opposed to Chisel developers), to use the latest release version of Chisel. This chisel-template repository is kept up-to-date, depending on the most recent version of Chisel. The recommended version is also captured near the top of this README, and in the Github releases section of this repo. If you encounter an issue with a released version of Chisel, please file an issue on GitHub mentioning the Chisel version and provide a simple test case (if possible). Try to reproduce the issue with the associated latest minor release (to verify that the issue hasn't been already addressed).

For more information on our versioning policy and what versions of the various Chisel ecosystem projects work together, see Chisel Project Versioning.

If you're developing a Chisel library (or chisel3 itself), you'll probably want to work closer to the tip of the development trunk. By default, the main branch of the chisel repository is configured to build and publish its version of the code as <version>+<n>-<commit hash>-SNAPSHOT. Updated SNAPSHOTs are published on every push to main. You are encouraged to do your development against the latest SNAPSHOT, but note that neither API nor ABI compatibility is guaranteed so your code may break at any time.

Roadmap

See Roadmap.