LFS example not working

[Sparkxxx]

Expected behavior

Obtain output as in docs example: https://nodemcu.readthedocs.io/en/dev/getting-started/#minimal-lfs-example

Actual behavior

node.flashreload("lfs.img")
Warning, deprecated API! node.flashreload. It will be removed soon. Use node.LFS interface instead. See documentation for details.
LFS region updated. Restarting.

print(LFS)
table: 0x3fff08a0 <<<--- In example we have NIL but I think this is here because of #define LUA_INIT_STRING "pcall(function() node.flashindex'_init'() end)"

node.flashindex("_init")()

print(LFS._list)
nil <<<--- this SHOULD NOT be NIL it sould return like in the example a table: 3fff0728

Test code

Starting from scratch just now 28-sep-2020 with
git clone --recurse-submodules https://github.com/nodemcu/nodemcu-firmware.git

BUILD NodeMCU
Modify user_config.h
#define LUA_NUMBER_INTEGRAL
#define LUA_FLASH_STORE 0x40000
#define LUA_INIT_STRING "pcall(function() node.flashindex'_init'() end)"
#define BUILD_SPIFFS
#define SPIFFS_CACHE 1 // Enable if you use you SPIFFS in R/W mode
//#define SPIFFS_MAX_FILESYSTEM_SIZE 0x20000
#define SPIFFS_MAX_OPEN_FILES 4 // maximum number of open files for SPIFFS
#define FS_OBJ_NAME_LEN 31 // maximum length of a filename

#define NODEMCU_EAGLEROM_PARTITION        1
#define NODEMCU_IROM0TEXT_PARTITION       2
#define NODEMCU_LFS0_PARTITION            3
#define NODEMCU_LFS1_PARTITION            4
#define NODEMCU_TLSCERT_PARTITION         5
#define NODEMCU_SPIFFS0_PARTITION         6
#define NODEMCU_SPIFFS1_PARTITION         7

#ifndef LUA_FLASH_STORE
#  define LUA_FLASH_STORE                 0x0
#endif

#ifndef SPIFFS_FIXED_LOCATION
  #define SPIFFS_FIXED_LOCATION           0x0
  // You'll rarely need to customize this, because nowadays
  // it's usually overruled by the partition table anyway.
#endif
#ifndef SPIFFS_MAX_FILESYSTEM_SIZE
#  define SPIFFS_MAX_FILESYSTEM_SIZE      0xFFFFFFFF
#endif
#define SPIFFS_SIZE_1M_BOUNDARY

docker run --rm -ti -v pwd/nodemcu-firmware:/opt/nodemcu-firmware marcelstoer/nodemcu-build build

Use NodeMCU PyFlasher to flash the image on ESP-12e

Command: esptool.py --port COM4 --baud 115200 --after no_reset write_flash --flash_mode dio 0x00000 D:\_NodeMCU LUA\dumps\nodemcu_integer_release_20200928-0829.bin --erase-all

esptool.py v2.6
Serial port COM4
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
MAC: 18:fe:34:f4:ed:44
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Erasing flash (this may take a while)...
Chip erase completed successfully in 9.0s
Flash params set to 0x0240
Compressed 659456 bytes to 443592...
Wrote 659456 bytes (443592 compressed) at 0x00000000 in 39.2 seconds (effective 134.6 kbit/s)...
Hash of data verified.

Leaving...
Staying in bootloader.

Firmware successfully flashed. Unplug/replug or reset device 
to switch back to normal boot mode.

Using ESPlorer v0.2.0
Formatting file system. Please wait...
..................................................
NodeMCU 3.0.0.0 built with Docker provided by frightanic.com
branch: release
commit: 64bbf00
release: 3.0-master_20200910
release DTS: 202009090323
SSL: true
build type: integer
LFS: 0x40000 bytes total capacity
modules: crypto,dht,file,gpio,http,mdns,mqtt,net,node,pwm2,rtctime,sjson,sntp,softuart,tmr,uart,wifi
build 2020-09-28 08:29 powered by Lua 5.1.4 on SDK 3.0.1-dev(fce080e)

----------------------------
No files found.
----------------------------
> 
Total : 3073746 bytes
Used  : 0 bytes
Remain: 3073746 bytes

Build LFS from lua_examples/lfs/
docker run --rm -ti -v pwd/nodemcu-firmware:/opt/nodemcu-firmware -v pwd/nodemcu-firmware/lua_examples/lfs/:/opt/lua marcelstoer/nodemcu-build lfs-image
Adding files: ./lfs_fragments.lua ./HTTP_OTA.lua ./_init.lua ./dummy_strings.lua
creating LFS_float_20200928-0840.img
creating LFS_integer_20200928-0840.img

Uploading to ESP file LFS_integer_20200928-0840.img...Success
>
----------------------------
LFS_integer_20200928-0840.img : 5670 bytes
----------------------------
Total file(s) : 1
Total size : 5670 bytes

Total : 3073746 bytes
Used  : 6024 bytes
Remain: 3067722 bytes

file.rename("LFS_integer_20200928-0840.img","lfs.img")
>
----------------------------
lfs.img : 5670 bytes
----------------------------
Total file(s) : 1
Total size : 5670 bytes

Total : 3073746 bytes
Used  : 6024 bytes
Remain: 3067722 bytes

Now using the guide https://nodemcu.readthedocs.io/en/dev/getting-started/#minimal-lfs-example
node.flashreload("lfs.img")
Warning, deprecated API! node.flashreload. It will be removed soon. Use node.LFS interface instead. See documentation for details.
LFS region updated. Restarting.

print(LFS)
table: 0x3fff08a0 **<<<--- In example we have NIL but I think this is here because of #define LUA_INIT_STRING "pcall(function() node.flashindex'_init'() end)"**

node.flashindex("_init")()

print(LFS._list)
nil **<<<--- this SHOULD NOT be NIL it sould return like in the example a table: 3fff0728**

Example is broken, maybe it's the lfs_addr, spiffs_addr ???

Create getinfo.lua in SPIFFS with content
print("The LFS addr is " .. node.getpartitiontable().lfs_addr)
print("The LFS size is " .. node.getpartitiontable().lfs_size)
print("The SPIFFS addr is " .. node.getpartitiontable().spiffs_addr)
print("The SPIFFS size is " .. node.getpartitiontable().spiffs_size)
> =node.heap()
43368
>
----------------------------
getinfo.lua : 264 bytes
lfs.img : 5670 bytes
----------------------------
Total file(s) : 2
Total size : 5934 bytes

Total : 3073746 bytes
Used  : 6777 bytes
Remain: 3066969 bytes

**> dofile("getinfo.lua")
Lua error: 	/opt/lua/_init.lua:97: attempt to call a nil value
stack traceback:
	/opt/lua/_init.lua:97: in function 'dofile'
	stdin:1: in main chunk
	[C]: ?
	[C]: ?**

With command line
>
print("The LFS addr is " .. node.getpartitiontable().lfs_addr)
The LFS addr is 663552
>
print("The LFS size is " .. node.getpartitiontable().lfs_size)
The LFS size is 262144
>
print("The SPIFFS addr is " .. node.getpartitiontable().spiffs_addr)
The SPIFFS addr is 925696
>
print("The SPIFFS size is " .. node.getpartitiontable().spiffs_size)
The SPIFFS size is 3256320

Tried to build also with default init.lua from SPIFFS //#define LUA_INIT_STRING "pcall(function() node.flashindex'_init'() end)" and the same thing happens, print(LFS._list) returns NIL
Tried building LFS with https://blog.ellisons.org.uk/article/nodemcu/a-lua-cross-compile-web-service/ with REMOTE LUAC.CROSS.INT (MASTER) but always get No zip file uploaded in Firefox and Chrome

NodeMCU startup banner

NodeMCU 3.0.0.0 built with Docker provided by frightanic.com
branch: release
commit: 64bbf00
release: 3.0-master_20200910
release DTS: 202009090323
SSL: true
build type: integer
LFS: 0x40000 bytes total capacity
modules: crypto,dht,file,gpio,http,mdns,mqtt,net,node,pwm2,rtctime,sjson,sntp,softuart,tmr,uart,wifi
build 2020-09-28 08:29 powered by Lua 5.1.4 on SDK 3.0.1-dev(fce080e)

Hardware

ESP-12e barebone

Thank you,

[Sparkxxx]

Digging forward so that I can use some LFS this is what I found

NodeMCU 3.0.0.0 built with Docker provided by frightanic.com
branch: dev
commit: 0a1d122
release: 3.0-master_20200910 +19
release DTS: 202009281842
SSL: true
build type: integer
LFS: 0x40000 bytes total capacity
modules: cron,crypto,dht,file,gpio,http,mdns,mqtt,net,node,pwm2,rtctime,sjson,sntp,softuart,spi,tls,tmr,uart,wifi
build 2020-09-29 08:43 powered by Lua 5.1.4 on SDK 3.0.1-dev(fce080e)

The documentation is outdated, please see the edits on the example:

node.flashreload("lfs.img") <<<--- should be node.LFS.reload("lfs.img")
-- flashreload() triggers a reset here.
print(LFS)
nil
node.flashindex("_init")() <<<--- although new functions apear related to LFS (node.flashreload got replaced by node.LFS.reload) node.flashindex("_init")() did not get a facelift so this is ok
-- LFS is now initialised.
print(LFS)
table: 3fff06e0
-- List the modules in the LFS.
print(LFS._list) <<<--- nil, correct call would be print(node.LFS.list()) => table: 0x3fff0918
table: 3fff0728
for k,v in pairs(LFS._list) do print(k,v) end <<<--- wrong due to LFS._list correct call would be for k,v in pairs(node.LFS.list()) do print(k,v) end
1 dummy_strings
2 _init
3 main
-- Call the LFS main() module.
LFS.main()
LFS main() module

So this brings us to the correct example:
node.LFS.reload("lfs.img")
LFS region updated. Restarting.

print(LFS)
nil

node.flashindex("_init")()

print(LFS)
table: 0x3fff04f0

print(node.LFS.list())
table: 0x3fff05a0

for k,v in pairs(node.LFS.list()) do print(k,v) end
1 HTTP_OTA
2 _init
3 dummy_strings
4 lfs_fragments

I don't know if it's just for the dev branch but my example now works producing the expected result.

First I will start by stating that I really appreciate your work, thank you all.
During the last week I've re-discovered NodeMCU lua (ignored so far due to being a new programming language, "every serios programmer" does C++ arduino) and after running out of ram I tried to use LFS. Lua and NodeMCU lua are new to me and the whole concept seemed promissing and accessible but got stuck in it due to the documentation which, although compelling does not tell you strait away what to do. It asumes that you know what you want to do and offers you the means to do it, which is ok if you are a Lua and embedded guru.

I propose a quick guide, more like a tour if you want for the serios programmer, who wants to know fast what this is all about without digging too much documentation. They'll get to it anyway but beying happy not frustrated, instant satisfaction guaranteed:
https://gist.github.com/Sparkxxx/08b58e547842f07a5294329a6b55d333

From my gist guide to be perfect I still have to find quick answers to:
1. nodemcu-firmware/local/fs - if files are put here the generated build for firmware will include them directly in LFS?
2. nodemcu-firmware/local/lua - if files are here they will be included in the generated SPIFFS?
!!! if above it's true then esptool can write multiple .bin files with one command so we only have to flash the board once to load firmware, SPIFFS and our program in LFS!!!
3. which is the best way (the correct numbers) to define LUA_FLASH_STORE, SPIFFS_SIZE so that OTA/Firmware upgrade will work

Thank you, hope this helps the project

[nwf]

Hey! Thanks for the quick-start guide prose. It would be great to get something like that integrated into the main documentation.

To your questions in reverse order:

• NodeMCU does not support full firmware OTA. We do support replacing LFS, with the caveat that LFS is undergoing heavy work across firmware revisions and we do not promise either backwards or forwards compatibility. If this is something you need, I think you will have to figure out how to wrangle it yourself. We would gladly consider patches to make this work.

• local/lua is not directly installed into LFS; instead, the files therein are built into local/fs/LFS.img. See the LFSimage target in tools/Makefile.

• local/fs is indeed used to generate the SPIFFS images (see the spiffsscript target in tools/Makefile), which land in files like bin/0xc0000-32mb.img which are ${flash_position}-${flash_size}.bin. In general, it's fine to use smaller SPIFFS images on larger flash devices, though I do not think SPIFFS will expand to fill free space without formatting from scratch.

All told, then, yes, you can land everything you want on the board in a single esptool.py flash command, but you will need to node.LFS.flash("LFS.img") at first boot. Probably the correct way to do this is to also land an init.lua containing the correct incantation in SPIFFS. (To avoid a reboot loop if something goes wrong with loading LFS, it may be useful to first test for LFS.img and then file.rename it to something else before node.LFS.flash-ing. At least in this case, you'll still have a Lua interpreter on the UART.)

[Sparkxxx]

Hi nwf,
I updated the howto with your answers and added some new questions and sections as things are getting clearer to me, I hope :).
Please check it and let's finish it if possible.
https://gist.github.com/Sparkxxx/08b58e547842f07a5294329a6b55d333

Thanks

[stromnet]

I've finally gotten LFS with latest 3.0 release running, but one thing from https://github.com/nodemcu/nodemcu-firmware/blob/dev/lua_examples/lfs/lfs_fragments.lua file that seems to be wrong is this one:

if node.flashindex() == nil then
  node.flashreload('flash.img')
end

This always returns nil, even if loaded, causing my device to get stuck in LFS reload boot loop until reflashing firmware.
node.flashindex('_init)) however returns a function, and if that is checked for instead, it's working..

Am I missing something, or is it just another thing missing in the docs?

[nwf]

That sounds like another docs/examples bug after the refactoring of LFS and attendant deprecation of node.flashindex(), I'm afraid. The generic test for LFS presence, regardless of which modules are in it, should probably be (I think; please check) if node.LFS.config() == nil then ... end, if you have the luxury of writing exclusively against the new API. Please feel free to open a PR catching the docs and/or examples up and sorry the examples aren't always up to snuff. (I hope that #2984 and #2983 help keep everything in sync in the future.)

[HHHartmann]

The plan was to leave the old interface working as it was. So this should probably get fixed if there is another release before we remove it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.