MIPS32 router: module_init not called for kernel module
buildroot, busybox, linux-kernel, mips, mips32
Solution
It turns out that the problem was related to a different kernel configuration between my development environment and the router. Specifically, my kernel was using `CONFIG_UNUSED_SYMBOLS` whereas the router's was not.
The reason this caused a problem even in a trivial module is that when the kernel loads a module it doesn't only look up the `module_init` symbol in the module's symbol table. Rather, it reads the `module` struct from the module (from the `.gnu.linkonce.this_module` section), and then calls the `init` module through that struct.
The offset of the `init` function pointer inside the `module` struct depends on the kernel configuration, which explains why the kernel can't find the `init` function if the configuration is different.
Thanks to Sam Protsenko for investing a lot of time in helping me crack this!
Problem
I'm developing a kernel module that I want to run on my router. The router model is DGN2200v2 by Netgear. It's running Linux 2.6.30 on MIPS. My problem is that when I load my module it seems that my `module_init` isn't getting called. I tried to narrow it down by modifying my `module_init` to return -3 (which indicates an error?) and `insmod` still reports success. I can see my module in the output of `lsmod`, but I don't see my `printk` output using `dmesg`. For starters, I wanted to create the simplest possible module: ``` #include <linux/module.h> #include <linux/kernel.h> #include <linux/init.h> static int my_init(void) { printk(KERN_EMERG "init_module() called\n"); return -3; } static void my_cleanup(void) { printk(KERN_EMERG "cleanup_module() called\n"); } module_init(my_init); module_exit(my_cleanup); ``` This is the Makefile I'm using: ``` TOOLCHAIN=/home/user/buildroot-2016.08/output/host/usr/bin/mips-buildroot-linux-uclibc- ARCH=mips CC = $(TOOLCHAIN)gcc KBUILD_CFLAGS:=. EXTRA_CFLAGS := -I/home/user/buildroot-2016.08/output/build/linux-headers-2.6.30/include\ -I/home/user/buildroot-2016.08/output/build/linux-headers-2.6.30/arch/mips/include/asm/mach-mipssim\ -I/home/user/buildroot-2016.08/output/build/linux-headers-2.6.30/arch/mips/include/asm/mach-generic\ -fno-pic -mno-abicalls -O2 obj-m := module.o KDIR := /home/user/buildroot-2016.08/output/build/linux-headers-2.6.30 PWD := $(shell pwd) default: $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules ``` I'm running `make` like so: ``` make ARCH=mips CROSS_COMPILE=/home/user/buildroot-2016.08/output/host/usr/bin/mips-buildroot-linux-uclibc- ``` which passes successfully. As you can see, I'm using Buildroot which I (hopefully) configured correctly. I can paste my .config if needed. I ran objdump on my module and didn't find a problem. In particular, the module_init symbol seems to point to the same place as my my_init function, and it seems to have the code I expect it to: ``` module.ko: file format elf32-tradbigmips module.ko architecture: mips:isa32, flags 0x00000011: HAS_RELOC, HAS_SYMS start address 0x00000000 private flags = 50001001: [abi=O32] [mips32] [not 32bitmode] [noreorder] MIPS ABI Flags Version: 0 ISA: MIPS32 GPR size: 32 CPR1 size: 0 CPR2 size: 0 FP ABI: Soft float ISA Extension: None ASEs: None FLAGS 1: 00000001 FLAGS 2: 00000000 Sections: Idx Name Size VMA LMA File off Algn 0 .MIPS.abiflags 00000018 00000000 00000000 00000038 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA, LINK_ONCE_SAME_SIZE 1 .reginfo 00000018 00000000 00000000 00000050 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA, LINK_ONCE_SAME_SIZE 2 .note.gnu.build-id 00000024 00000018 00000018 00000068 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 3 .text 00000040 00000000 00000000 00000090 2**4 CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE 4 .rodata.str1.4 00000038 00000000 00000000 000000d0 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 5 .modinfo 0000005c 00000000 00000000 00000108 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 6 .data 00000000 00000000 00000000 00000170 2**4 CONTENTS, ALLOC, LOAD, DATA 7 .gnu.linkonce.this_module 0000014c 00000000 00000000 00000170 2**2 CONTENTS, ALLOC, LOAD, RELOC, DATA, LINK_ONCE_DISCARD 8 .bss 00000000 00000000 00000000 000002c0 2**4 ALLOC 9 .comment 00000040 00000000 00000000 000002c0 2**0 CONTENTS, READONLY 10 .pdr 00000040 00000000 00000000 00000300 2**2 CONTENTS, RELOC, READONLY 11 .gnu.attributes 00000010 00000000 00000000 00000340 2**0 CONTENTS, READONLY 12 .mdebug.abi32 00000000 00000000 00000000 00000350 2**0 CONTENTS, READONLY SYMBOL TABLE: 00000000 l d .MIPS.abiflags 00000000 .MIPS.abiflags 00000000 l d .reginfo 00000000 .reginfo 00000018 l d .note.gnu.build-id 00000000 .note.gnu.build-id 00000000 l d .text 00000000 .text 00000000 l d .rodata.str1.4 00000000 .rodata.str1.4 00000000 l d .modinfo 00000000 .modinfo 00000000 l d .data 00000000 .data 00000000 l d .gnu.linkonce.this_module 00000000 .gnu.linkonce.this_module 00000000 l d .bss 00000000 .bss 00000000 l d .comment 00000000 .comment 00000000 l d .pdr 00000000 .pdr 00000000 l d .gnu.attributes 00000000 .gnu.attributes 00000000 l d .mdebug.abi32 00000000 .mdebug.abi32 00000000 l df *ABS* 00000000 module.c 00000000 l F .text 0000002c my_init 0000002c l F .text 00000014 my_cleanup 00000000 l .rodata.str1.4 00000000 $LC0 0000001c l .rodata.str1.4 00000000 $LC1 00000000 l df *ABS* 00000000 module.mod.c 00000000 l O .modinfo 00000023 __mod_srcversion23 00000024 l O .modinfo 00000009 __module_depends 00000030 l O .modinfo 0000002c __mod_vermagic5 00000000 g O .gnu.linkonce.this_module 0000014c __this_module 0000002c g F .text 00000014 cleanup_module 00000000 g F .text 0000002c init_module 00000000 *UND* 00000000 printk Disassembly of section .MIPS.abiflags: 00000000 <.MIPS.abiflags>: 0: 00002001 movf a0,zero,$fcc0 4: 01000003 0x1000003 ... 10: 00000001 movf zero,zero,$fcc0 14: 00000000 nop Disassembly of section .reginfo: 00000000 <.reginfo>: 0: a2000014 sb zero,20(s0) ... 14: 00007fef 0x7fef Disassembly of section .note.gnu.build-id: 00000018 <.note.gnu.build-id>: 18: 00000004 sllv zero,zero,zero 1c: 00000014 0x14 20: 00000003 sra zero,zero,0x0 24: 474e5500 c1 0x14e5500 28: c8e5d654 lwc2 $5,-10668(a3) 2c: cb477d3d lwc2 $7,32061(k0) 30: dfa48d71 ldc3 $4,-29327(sp) 34: c2ea16da ll t2,5850(s7) 38: f6bcae7d sdc1 $f28,-20867(s5) Disassembly of section .text: 00000000 <init_module>: 0: 27bdffe8 addiu sp,sp,-24 4: 3c040000 lui a0,0x0 4: R_MIPS_HI16 $LC0 8: 3c020000 lui v0,0x0 8: R_MIPS_HI16 printk c: afbf0014 sw ra,20(sp) 10: 24420000 addiu v0,v0,0 10: R_MIPS_LO16 printk 14: 0040f809 jalr v0 18: 24840000 addiu a0,a0,0 18: R_MIPS_LO16 $LC0 1c: 8fbf0014 lw ra,20(sp) 20: 2402fffd li v0,-3 24: 03e00008 jr ra 28: 27bd0018 addiu sp,sp,24 ``` `modinfo` output also matches what I expect (same `modinfo` output as for another .ko that's found on the router, except for the `srcversion` which my module has but the other module on the router doesn't): ``` filename: /home/user/module/module.ko srcversion: B0BADBA395A121CF49B74DC depends: vermagic: 2.6.30 mod_unload MIPS32_R1 32BIT ``` It's entirely possible that I messed something up in my Buildroot configuration, or something doesn't quite match the CPU type of the router, but my init code is so minimal that I'm out of ideas as to what could be wrong.