Info:
A documentation of technical learning and development practice. Notes on discoveries accumulated through engaging with research, debugging, problem-solving, and systems implementation. The insights and knowledge that emerge from hands-on technical work.
MLSys 2: Computation Graphs, Training Systems, and Distributed Scaling
LLM Quantization on RTX 4060: Where 4-bit Decode Falls Behind FP16

MLSys 1: Hardware, Memory, Parallelism, and Data Layout
Notes on Programming Language Design and Implementation
This set of notes is compiled from Programming Language Pragmatics course review material, covering type systems, composite types, subroutines, object orientation, functional languages, concurrency, compilation and runtime systems, and other topics.
The focus is on placing programming language concepts back into a few core questions:
- How is a value classified, checked, and interpreted?
- How do complex data structures map to memory?
- What actually happens at runtime during a function call?
- How are objects, inheritance, and dynamic dispatch implemented?
- How do functional languages organize computation?
- How do concurrent programs manage multiple control flows and shared state?
- How do compilers, virtual machines, and runtime systems support high-level language abstractions?
Programming …
07 Building and Running Programs
The core question of this chapter is: how does a piece of source code become a runnable program, and what management work must the language system continue to perform while the program runs.
The preceding chapters discussed language features: types, objects, functions, concurrency, memory. Here the focus is on how those features land:
source code
-> front end
-> intermediate form
-> optimization
-> code generation
-> executable / bytecode
-> run-time system / virtual machine
-> execution
This chapter can be divided into two halves:
- Building a runnable program: how the compiler analyzes, optimizes, and generates code.
- Run-time program management: how the runtime system, virtual machine, and JIT support program execution.
Together they answer one question: how the …
06 Concurrency
The core question of the concurrency chapter is: when a program has multiple control flows advancing simultaneously, how do the language and runtime organize them, schedule them, and prevent them from mutually corrupting shared state.
The execution order of single-threaded programs is usually relatively direct: step by step forward, with state changes relatively easy to trace. The difficulty of concurrent programs lies in this: the execution order of multiple tasks may interleave, and this interleaving order is not fully controlled by the programmer.
So concurrent programming is not just about “how to make programs faster,” but also includes:
- How multiple tasks advance simultaneously
- Which tasks truly run at the same time
- How shared data is protected
- How waiting conditions are …
05 Functional Languages
The core question of the functional languages chapter is: what happens to programming languages when “functions” are placed at the center of the language, rather than “state modification” at the center of the program.
In imperative programming, programs are typically understood as a sequence of commands:
Change variables
Update state
Execute loops
Modify objects
Functional programming is more concerned with expressions, function composition, value transformations, and referential transparency. It cares about:
- Whether functions can be passed around like ordinary values
- Whether data can remain immutable
- Whether an expression can be replaced by its value
- Whether evaluation order affects results
- Whether function calls can be cached
- Whether code itself can be treated as …
04 Object Orientation and Dynamic Dispatch
The core question of the object orientation chapter is: how do languages bind data and operations together, and how does the same piece of code execute different logic at runtime based on the object’s actual type.
On the surface, OOP is about class, object, inheritance, method – these syntactic elements. At a deeper level, it’s concerned with several things:
- How to encapsulate state
- How to hide implementation details
- How to reuse existing code
- How to reference subtype objects through supertype references
- How to decide which method to call at runtime
- How to initialize and destroy objects
- How to implement interfaces, abstract classes, dynamic dispatch, and vtables
So this chapter is not simply “object-oriented programming philosophy,” but how the object model …
03 Subroutines and Control Abstraction
The core question of the subroutines chapter is: how does a language encapsulate a block of code into a callable unit, and what actually happens at runtime during a function call.
On the surface, a function call is just:
f(x)
But at the implementation level, many things must be handled:
- How parameters are passed in
- How return values are passed back
- Where local variables are placed
- How to return to the original location after the call ends
- How nested functions access outer variables
- How to clean up stack frames when exceptions occur
- How coroutines and threads save and restore control flow
So this chapter connects “function abstraction in the language” with “control flow and stack frames on the machine.”
Calling Sequence
The calling sequence is a set of steps that …
02 Composite Types and Memory Layout
The core question of the composite types chapter is: how do languages organize multiple values into more complex data structures, and how are those structures actually represented in memory.
If the Types chapter focuses on “what a value is,” the Composite Types chapter focuses on “how multiple values are placed together.” This is not just a syntax design issue; it also directly affects memory layout, access efficiency, safety, assignment semantics, comparison semantics, and garbage collection.
Think of this chapter as a set of connections between the language layer and the machine layer:
- record / struct: how multiple fields are arranged
- union / variant record: how the same block of memory represents different forms of data
- array / slice: how contiguous data is …
01 Type Systems
The core question of the type systems chapter is: how does a language understand “what” a piece of data is, what operations it can perform, when should errors be discovered, and how can code be safely reused across different data.
Types can be thought of as a constraint system within a programming language. On one hand, they help programmers express intent; on the other, they help the compiler or runtime discover invalid operations. If a value is seen as int, it can participate in integer arithmetic; if seen as string, it can perform string concatenation, indexing, matching, etc. The same binary data, interpreted under different types, carries completely different meanings.
The Role of Types
Types serve several purposes in programming languages:
- Tell the compiler or runtime …