Ethernet in a cold climate / SMDK6410

Hi -

I broke out my old SMDK6410 last night and tried linux-next then Ben's git on it (currently next-s3c64xx-updates) started from s3c6410 Qi / SD Card boot.

It's basically happy but OTG doesn't seem workable for Ethernet gadget.
Well it's dumping a lot of debug messages so I guess maybe that's
where it got up to, fair enough.

I eyeballed the back of the SMDK and I see a CS8900A and a SMSC LAN9115 both sitting there, coming out on separate RJ45s. When I look at the machine init file, I see Mark added

static struct resource smdk6410_smsc911x_resources[] = {
	[0] = {
		.start = 0x18000000,
		.end   = 0x18000000 + SZ_64K - 1,
		.flags = IORESOURCE_MEM,
	},

...

What is the deal with that in terms of SDMK settings and assumptions about bootloader prepping nCS signals? SMDK docs I have let me twiddle switches and get "ethernet" connected to various selects. What does 0x18000000 represent and how should the board be set so CS8900A doesn't crash the party?

After some debug-by-printk, I see smsc911x probe is failing with:

smsc911x: BYTE_TEST: 0xFFFFFFFF
smsc911x: top 16 bits equal to bottom 16 bits
smsc911x: This may mean the chip is set for 32 bit while the bus is reading 16 bit
smsc911x: init failed

So I am guessing the nCS pin is not configured, something is missing a clock... one of the dozens of mini DIP switches is "wrong"?

-Andy

I've never used the board with the CS8900A enabled, the bootloaders I've seen appear to support the SMSC chip only.

The memory address was lifted from the Samsung BSP code IIRC.

On 12/28/09 19:00, Somebody in the thread at some point said:

Hi Mark -

Thanks for the quick reply.

It's fine if I get the SMSC chip happy, I just mention the CS8900A because it is there are the docs only talk about "ethernet" so I expect conflict.

So what was your setup when Ethernet worked OK...

- the Samsung U-Boot fork that came with the SMDK?

- magic address 0x18000000 OK

- 60+ minidip config was what?

I'm guessing 0x18000000 is on a particular nCS signal that needs configuring (should that really not be the business of the machine definition file?) and the mini DIPs setting to match.

-Andy

Yes, whatever was flashed on it when Samsung last had the board. The bootloader does explicitly say it's looking for the SMSC chip when setting it up.

Yes.

Factory default - I can't immediately visualise which switches you're talking about and I'm out of the office until next Monday so I can't check with the boards how they're set up. I don't recall any relevant switches.

I'd expect so.

I'm not sure what you mean by the "machine definition file" here? The large number of non-probable and incompatible changes on the Samsung boards mean that it's generally best to leave them at factory default settings unless you have a particular need to change things since the kernel tends to be configured for the default setup.

On 12/28/09 19:22, Somebody in the thread at some point said:

Hi -

I went and looked at the sources for that I got from Samsung, it talks only about CS8900.

#define CS8900_Tacs (0x0) // 0clk address set-up
#define CS8900_Tcos (0x4) // 4clk chip selection set-up
#define CS8900_Tacc (0xE) // 14clk access cycle
#define CS8900_Tcoh (0x1) // 1clk chip selection hold
#define CS8900_Tah (0x4) // 4clk address holding time
#define CS8900_Tacp (0x6) // 6clk page mode access cycle
#define CS8900_PMC (0x0) // normal(1data)page mode configuration

...

	SROM_BW_REG &= ~(0xf << 4);
	SROM_BW_REG |= (1<<7) | (1<<6) | (1<<4);
	SROM_BC1_REG = 

((CS8900_Tacs<<28)+(CS8900_Tcos<<24)+(CS8900_Tacc<<16)+(CS8900_Tcoh<<12)+(CS8900_Tah<<8)+(CS8900_Tacp<<4)+(CS8900_PMC));

I had a quick grep for corresponding register defines in S3C64xx Linux bits but didn't see anything.

No worries. There are a row of mainly 4-switch mini-dips under the LCD that control who gets to see which chip select, I'm pretty sure setting them wrong will break access to "ethernet" whatever that means.

If we had like S3C6410_PHYS_NCS2 instead of 0x18000000 I could get a clue, but I realize you didn't get that from the Samsung tree either.

I meant mach-smdk6410.c, since it wants to configure Linux to use stuff that has dependencies it doesn't control. If it's going to claim something is there at 0x18000000 for a particular machine then it's at least arguable it should configure any soft prerequisites (nCS) accordingly and document the ones out of its control.

Yeah. But again if the successful operation of the file that defines how Linux works with the SMDK is dependent on stuff like magic switch configuration ideally it would tell what the dependencies are in a comment, accepting random changes to boards you don't have are out of scope. That one file anyway mach-smdk6410.c is super-specific to the board and is the only place to talk about the dependencies.

I understand that you might be able to get that working and yet not understand the intricacies of the dev board, and we're still ahead that you managed to get anything working.

My board is as shipped with the switches AFAIK except configured for SD Card boot according to Samsung docs.

-Andy

That's... interesting. To be honest I don't think I've ever seen the source code for the bootloaders, but none of the SMDK boards I've had have defaulted to using the CS8900 - I've never actually seen software that talked to it to my knowledge, the Samsung bootloaders and kernels were all using the SMSC chip and obviously that's what mainline has been doing too.

Oh, sure - you were saying it should configure stuff and obviously most of these things are completely out of scope for software which wasn't very clear at all. I quite frankly wasn't aware that there were any relevant soft controls at all.

By the way, the other thing to watch out for with the Samsung boards is that as well as the DIP switches there's a bunch of resistor fit options that can be changed as well.

Right, and hopefully this is the sort of thing that Samsung are going to be doing with their mainline work.

It sounds like you've at least got a completely different bootloader (or the source is highly misleading). The other thing to check is which rev of the board you have, there have been several revisions though I don't recall any changes in the networking.

On 12/28/09 20:21, Somebody in the thread at some point said:

Hi -

Well it's a fact. I sent you a link these GPL'd sources separately.

I would suggest that relying on (mysterious, changeable) bootloader to set up stuff like nCS timing and which nCS would be a dependency that would be good to avoid. People are trying to use the dev board as a basis for their own designs and they have to ship an actual non-quantum-mist bootloader with that.

Yeah, I saw it on SPI0 before (I want to use this as an SPI peripheral meddling platform at the moment). The socket is sitting there with NC resistor pack so no signals.

Right I wrote Qi bootloader support for s3c6410. This allows true SD boot from SMDK. "True SD boot" means that the bootloader itself is on SD Card and is brought over into RAM by the CPU ROM, so there is nothing to brick on the device itself.

If anyone is interested see the txtr branch of

http://git.warmcat.com/cgi-bin/cgit/qi/

SMDK6410 docs tell the following to enable SD Card boot:

CFGB1 - CFG4 (under the LCD)

~____ __~_ ~___ ____ ____ ____ ~_ ~~~_ __~_ ____ ____

CFG3

[
[
[
[
[
]

CFG4

]
]
[
[

CFG1

]
[
[
[

J8: 1-2
J7: 1-2
J6: 1-2

You also need a particular ROM revision in your SMDK but this one is over a year old so I guess it's common now.

-Andy

Like I say, I wasn't aware that these were soft configurable in the first place - I'd never needed to look beyond the hookup of the device in the Samsung kernel plus some detective work with the IRQ polarity configuration in the driver since the board miswires it. I do agree that if these things are software configurable then we ought to be doing it in the kernel, I strongly suspect that any setup done by the bootloader is only being done when booting over the network.

Qi would be a massive step back for me in terms of usability unfortunately - having to transfer things to an SD card to boot them would add a lot to my edit, compile run cycle compared to netboot.

On 12/28/09 22:44, Somebody in the thread at some point said:

Hi Mark -

Glad we agree on that.

Bootloaders are kind of out of control of these dev boards, iMX31 LiteKIT even came with a closed source bootloader. That destabilizes any certainty you can have about basing a design on the dev board, because you don't know how dependent successful behaviour will be on magic hidden in the bootloader, even if you got a fully working software stack on the dev board.

So I think the mach-blah.c files should try to assert the state of dependencies themselves for stuff they are saying they support, and make no assumption that the bootloader has done anything. (And indeed I don't think the bootloaders should do anything except enough to get Linux started, but that's another story).

Like I say it sets up a magic nCS signal in the bootloader init, it talks about it as CS8900A but elsewhere they talk about "ethernet", so it may simply use this nCS to get to both ethernet chips, I didn't look at the schematic yet.

Just a sanity check, it was the case for you too that g_ether is broken and although you get a logical usb0 at both sides, they cannot communicate?

You seem pretty sure about that, so much so I guess you never tried this workflow.

There are two cases with SD Card boot and kernel work:

1) you screwed your kernel and it blows chunks on boot. In that case, you either have to make arrangements in the bootloader to check a button and use a backup kernel if pressed (Qi supports this generically), or pop the SD card and write the kernel on the host. In the button case, you just reset with that button down and use a host build script to ssh / rsync over your next kernel try without touching the SD Card.

I guess you see the above case more often than most because of what you're doing, but actually I find it's only when I meddle in particular areas I run that risk; most things fail to work nicely until fixed or OOPS in a way that you can still send over your fixes in that session.

2) you are working on a driver that can be done modular while there's risk of it blowing chunks. In this case your build script can build the kernel on the host and update the module over ssh / rsync. You then modprobe -r / modprobe it without touching the SD Card or even resetting.

I don't see those workflows as any "massive step back"; compared to raw NAND being able to pop and rewrite the SD card in an emergency is certainly a massive step forward.

There's another reason to not act special towards the device with stuff that does not represent what will ship, as the developer you're the first user of it and if you're not hammering on, experiencing and optimizing the boot flow that ships, nobody else will.

In addition if opposed to network boot case you are working on an SD image all the time that can be taken as the shipping image, not some special tree of files on a host whose speed of access and code paths to get to it are completely different.

-Andy

I'd assume so given that the board has a physical switch selecting between the two ethernet controllers.

I've never used g_ether, and my use of USB has been limited to testing power path management.

Really, I've tried this and similar workflows. Network boot beats everything else I've tried.

Either physically swapping the SD card or having to dance with the bootloader to revert to a known good configuration then refresh are very involved relative to flipping the power switch and booting, partly because they do actually take a relatively long time and partly because they aren't the same routine normally used to boot and so require more attention and thought (not much, but enough).

This is what I'm actually doing - things that can be built modular generally are, but there's always things that won't allow that.

Compared to NAND removable media can be a win, yes, though with fast JTAG having to rewrite the flash needn't be any slower so it's not quite that clear cut.

With what I do it is relatively rare for me to work directly on production systems - the majority of the systems I'm using for driver development involve lots of flying wires and never see the light of anything except my desk, the end result being either core kernel changes or chip-specific drivers. Where I am working on production systems generally either it's not possible to do anything except JTAG onto the flash or the only thing changed by netboot is where the kernel image is loaded from.

On 12/29/09 12:33, Somebody in the thread at some point said:

Hi -

Yeah I found that thismorning, CFG6. I added a comment about that and the (discovered by trial and error) requirement that it's nCS1 specifically needed to drive it to work with the 0x18000000 magic number.

I found that the nCS1 bus width defaults to 8-bit in the CPU, otherwise it's now addressable. So I am adding some constants for s3c64xx SROM unit in the platform stuff and will set it up in mach-smdk6410.c.

Yeah. All I can say is that constant testing of kernels that fail to boot is a somewhat specialized case. You would be into pressing a button or popping the card.

Sure. In the ones you build modular, there's no cost to the SD flow.

Right, but if you can eliminate JTAG in the flow, especially at the factory, that is a major simplification.

SD + Qi has the advantage it eliminates completely "board bringup" at the factory. Qi has no state stored on the board that changes boot flow. So you can give a virgin board an SD and it brings itself up with no special gear or processing steps.

Yeah understood.

Again all I can say is it's specialized case. For normal dev work SD is either painless or hugely advantageous.

-Andy

An awful lot of production setups are capable of handling JTAG already so don't have much of an issue here. Certainly as a developer my preference here would be for fast JTAG over SD card - it's at least as fast, there's only one system involved and it works just the same no matter what the state of the system is.

Assuming the hardware can cope with it, and there's component cost, board area and mechanical concerns to address before it gets designed in. There are a lot of systems where it would be useful but there's drawbacks you have to bear in mind when pushing it.

On 12/30/09 13:16, Somebody in the thread at some point said:

Hi -

If you have JTAG gear that works reliably (I never had much luck with OpenOCD, I only ever got it to work with the OM debug board with one specific build of a library, not the latest one either). I have a Wind River JTAG dongle here that only works on Fedora 6 or something, it's not really workable here or at a factory to set up VMs to please this piece of test gear. I guess you have better luck depending on JTAG than I have, it certainly creates a burden to use it in my experience.

Actually with SD basis you simply remove all other mass NV devices from your board. SD cards can be duplicated out of the factory so there's no real comparison needed between raw NAND bringup over JTAG and SD Card basis.

The only piece of "board bringup" needed is to allocate and write board identity data somewhere like MACs if you want that to live on the board and not the SD, and that can be done from the test SD image acquiring the data over the network.

Customers don't have JTAG gear either so those arrangements and that work are specific to the production action. However customers do have SD readers and that can be part of a debricking scheme that does not involve product return.

I don't really regard those as drawbacks.

Component cost is just the connector and four pullups, I guess it's $0.70.

Yeah you need to make the uSD connector accessible if you remove a cover, that's generally not hard.

CPUs in the class where it is useful (ARM11+) all tend to have one or more SD peripheral unit on the CPU already, so that's for free.

With true SD boot, cost of the SD card itself is balanced by being able to remove all

- NAND

- NOR

- JTAG connector + passives

from the board, and all production and test actions around those now they are gone.

Space for the uSD socket on one side of the board or another is less than typical footprint of NAND and JTAG connector that used to be on the board.

I think you find SD basis is normally for free or cheaper.

There are a bunch of horrors around working with raw NAND that also disappear now your access to NAND is "cooked" by the controller on the SD Card. You no longer need to deal with ECC, BBT, jffs2 and the like but can use normal filesystems directly.

So I think the issue is more the tendency to not bear the drawbacks of existing schemes and bootloaders in mind when considering different workflows, since dealing with the issues from those are what people are used to.

-Andy

The one I've had the most positive experience with was the BDI2000 - no host software required, it's controlled over the network via telnet (and possibly other things now) for non-debugging activity and provides a GDB stub for debugging. Like everything else there are plenty of rubbish products out there.

That 70 cents does rather add up once you start to ship in volume, even a fraction of that would.

In some form factors. In others it's very problematic.

Assuming you've not used all the pins for something else, which is again a problem for some classes of device.

On 12/31/09 12:27, Somebody in the thread at some point said:

Hi -

Sounds pretty good, thanks for the tip.

You did notice that you get to save money back from removing NAND and JTAG connectors?

Due to the other advantages of SD Boot basis you can also expect to get your product to market quicker, spend less time in the factory, and have less returns all of which save money.

So I wouldn't let this notional $0.70 put you off.

Dunno how problematic it can be actually, unless you dunk your board in epoxy. uSD connectors are very small footprint and low profile since they're widely used in consumer equipment like phones. Since you only need access to it in a crisis due to bricking it, it's OK if you have to remove a cover or whatever. Space shouldn't be an issue due to what it allows you to throw out from the board. I'm sure there are special cases but I maintain "generally, that's not hard".

Yeah it's true, if the mux arrangements conflict with other peripheral units you must have something's got to give.

But many chips have 2 or 3 SD units, mux arrangements that bring the same things to different pins. In the case of SD interface, it's a given in context of phones where a lot of these chips go, so you tend not to find other IO on those balls that is critical and does not also appear on other ball muxes. (It's only 6 balls anyway, it's not like a memory bus). It seems you can typically expect you're going to be able to use SD unless you find a specific conflict because the chips are deliberately designed to enable SD usage in typical design.

-Andy

I pine for the BDI2000 I was lent...

Of course, with bed of nails type systems it is probably possible to allow the system to boot off what you like, NAND is easily ~18 connections to the board, SD is of course less.

A lot of times these are done as 'bed of nails' pads on the PCB so hardly a bank breaker... Also, see PoP argument below.

And as shown by the nook, make it really easy for your users to root the thing without trying. You've been spoiled a bit by having an open design to play with, not everyone likes their users messing with the hardware once it has left the factory.

Also as another cost item, a lot of the newer cpus with PoP style memory have some form of NAND/OneNAND built into the system already so adding an extra SD card is often seen as a waste of extant storage.

A lot of the time, the bigger manufacturers have enough money that a commercial JTAG system makes sense, as not only can it program the board but there are also connectivity checking systems that allow you to do signal connectiivty testing and other such wonderfullnesses for you.

Smaller vendors tend to cobble together something using open-source tools, etc.

and of course, a lot of modern mobile devices need SD for talking to other devices, such as WLAN chips, external storage cards, etc. It can soon take a toll on what is available.

On 01/04/10 16:26, Somebody in the thread at some point said:

Hi -

Yeah... all kinds of complex board bringup schemes are possible... but... are they wise? It's special for the factory and special for the pcb revision. It's extra cost and management.

For example Openmoko went your NAND bringup route and it was basically an intricate, complex Hell that few people understood and fewer, like zero, actually wanted to go to the factory with. OpenOCD basis turned it into a lottery if even internal devs could work with it.

For smaller runs test jigs are popular, they have the same problem of requiring developer and design effort.

Often when I have this conversation, it mainly runs into the other guy having heard / decided that "real men" must have jigs / jtag / u-boot or whatever it is. But actually, it can be a huge amount of dead weight around small to medium run development effort and it can be entirely optional to tie those stones around your neck.

Maybe you like sorting that out but I really prefer the SD Boot scheme where there is no factory bringup arrangement at all except the shipping rootfs with some bash scripts turning a board from the oven into a shipping device.

You're referring to NAND programming here I assume.

Something has to be on the other end of those spring probes / jig and it has to be prepared / maintained / tested. The burden does not end with the fact you made contact with some probes. It's special for the factory and cannot be repeated by the user (or even the devs without grief). If you make another run two years later and the guy who did the magic left, maybe the processing steps to turn an image into SVF is lost and suddenly the visit to the factory is a risk.

A better argument for that I have heard would be, "well, I have to 100% test connectivity on all my external ports, don't I, spring probes will solve it". But this argument is also broken, you are not testing connector function that way.

If it's doable on the design, the optimal test strategy is often loopback plugs and selftest, although how far you can get with that depends on exactly what your device exports on the connectors.

No, that does not have to follow.

iMX31 has hard crypto chain of trust all the way from ROM, through SD bootloader being signed, which can then validate signatures of kernel and rootfs. If you want it locked up and do true SD boot you can do it.

Further you are mistaken if you think having a NAND on a board is any impediment to getting / changing its contents to someone who wants to do that. It makes it harder to change the contents without swapping the chip, but that's a double edge sword in debrickability terms when you only targeted a factory with a jig to be able to do that.

And modchip folks will tell you that once you have the rootfs, you stand a decent chance of finding something that may let you circumvent security and reflash it without NAND lifting.

You need a bit more of a story than you just put the same stuff in a NAND on your board and it's Fort Knox now.

Some do PoP and some don't; the mix of assets in the PoP is soft if you are a large vendor, so it's possible to just get DDR in there. DDR is the thing where PoP actually delivers something, since it removed the board layout and verification burden.

Yeah. But many (most?) folks working on this stuff are trying to bring complex products to market with insufficient resources and pressed for time. There isn't any internal test team. What there is mainly is the choices that the developer will make as to how to come at production.

By smart choices about how to

- boot

- create the rootfs

- test the device

the developers can eliminate several kinds of downstream steps that are currently considered unavoidable and get the optimal situation in previously dangerous areas like board bringup and rootfs maintainence.

Sure, if you need more SD units than the chip has, something's gonna have to give for sure. But I look around and most things I see don't have 3 SDs used up.

The last few objections have basically been, "what if I can't use SD Boot for some reason? Where is your God NOW???"

If you actually can't use it because you dip your board in epoxy; you need all the SD ports for something else; your CPU doesn't have SD boot; your customer thinks a NAND on a PCB is secure... then don't bother considering what you can't use and do something else :-)

Otherwise, you should study carefully the advantages you can get from true SD Boot in simplifying the hardware, software engineering and production / test / debrick. It has pretty revolutionary benefits.

-Andy

Look, it's very obvious from every one of your replies that you think the sun shines out of your SD cards, it's superiour to whatever else people are using, and that everyone should use it.

You have an answer for every argument to the contary.

Well, fine, you're entitled to that opinion.

Now can we stop discussing this and move on please.

Yeah, that's normally considered a good sign for a thesis.

Sure.

However if anyone else is interested in help setting up with it, feel free to contact me direct.

-Andy

On Mon, 04 Jan 2010 18:09:26 +0000
Andy Green <[email protected]> wrote:

Hi Andy,

I was trying to access your Qi tree
(http://git.warmcat.com/cgi-bin/cgit/qi/), without any success (it times out after a while...).

Is this intentional?

Thanks,

	M.

Fast. Cheap. Reliable. Pick two.

Fwiw:

1. 100% of CPUs I've used don't have SD boot.
2. I don't want to fiddle about with removing and replacing an SD card every time I change the kernel / bootloader.
Give me a cable and computer!
3. It's good to have a debricking strategy, but even better if it's the same as or similar to the regular development strategy - better test coverage, less knowledge to get lost.
4. Thus hardware USB boot is surely best, and TFTP the next best :-)
5. JTAG board connectivity is sometimes done with Perl and parallel ports too; it's not reserved only for expensive dev packages.

-- Jamie