compiler dependency does not give access to compiler

[arnetheduck]

.nimble:

requires "nim >= 1.6.16",
         "compiler"

nimble setup -l
   Warning: Using project local deps mode
      Info: using /home/arnetheduck/src/nimph/nimbledeps/pkgs2/nim-2.0.0-35b9d926d314ce8c3326902c138f5c2247cfd390/bin/nim for compilation
  Verifying dependencies for nimph@0.1.0
     Info:  "nimble.paths" is updated.
     Info:  "config.nims" is already set up.

nimble.paths:

arnetheduck@praeceps:~/src/nimph$ cat nimble.paths
--noNimblePath

notably, compiler path is missing (compiler/ is not part of nim standard paths)

[daylinmorgan]

Related to #1163

[jmgomez]

Should we treat it as a some sort of "special" package? As far as I can tell is defined as:

  {
    "name": "compiler",
    "alias": "nim"
  },

Without knowing much about nimble, all I can say is either we expand the definition or we treat compiler differently

[daylinmorgan]

Nimble is translating that to the nim package nimpkgs link instead of packages.json.

Which when using either nimble install nim or nimble install compiler is using a before install task defined in the nim.nimble in the main repo to run the build script. But ignores it as requires "compiler" in a .nimble.

Being able to access the "compiler" as a source itself should really only be necessary if you need a version separate from the compiler you are using. Since imports can reference the compiler with import "$nim" / compiler.

We could either package the compiler source as a standalone repo that would be importable with import compiler and it's own compiler.nimble that doesn't build the nim binary or tools. Or modify nimble to do this behind the scenes and treat the compiler dependency special rather than as an alias for nim.

I would say you we could do something with https://github.com/nim-lang/nim/subdir?=compiler and put the compiler.nimble there Oh wait this already exists!

But the current compiler.nimble isn't usable with nimble, since nimble only copies the subdir to the nimbleDir.

But if we change it to include "$lib" instead of include ../lib. Then the following seems to work for using the compiler as a package.
dummy.nimble

version       = "0.1.0"
author        = "Daylin Morgan"
description   = "A new awesome nimble package"
license       = "MIT"
srcDir        = "src"
bin           = @["dummy"]

requires "nim >= 2.0.2"
requires "https://github.com/daylinmorgan/nim?subdir=compiler#compiler-nimble-fix"

src/dummy.nim

import compiler/ast

when isMainModule:
  echo("Hello, World!")

So if we make a PR to nim-lang/nim to change the compiler.nimble and update the packages.json to treat the compiler as a package pointing to the subdir then it would work. Though the changes to packages.json should probably coincide with the first nim version release containing the updated compiler.nimble

[arnetheduck]

Should we treat it as a some sort of "special" package? As far as I can tell is defined as:

preferably no - the majority of the code therein can be reasoned about as if it were a package like any other.

Being able to access the "compiler" as a source itself should really only be necessary if you need a version separate from the compiler you are using.

there's no particular reason to treat the majority of code in the compiler in any special way.

iff the compiler needs to have a version constraint on the standard library for whatever reason, it should express that constraint in its declaration of requirements, and those parts should ideally be separated into their own package (nim-lang-core or whatever).

It is not a given that I want the same version of compiler as the source code is being compiled with - ie if I want to access the source code parser of the compiler as it worked in nim 1.6, I should (ideally) be able to access that version regardless of which nim version I'm using to compile my application. The $nim/... is a hack at best because it overconstrains the versioning requirements and leaves the natural flexibility that would come from not special-casing all of the compiler on the table.

This follows from a much broader point: the code in the compiler is / can be treated as a "library" and follow the rules of any other library (and just like some libraries come with "supporting binaries", so can the nim compiler be treated as a supporting binary (ie think c2nim, parser generators and other similar transformation tooling).

The same applies to the standard library: it is just one more nim library and large parts of it need no special version of the language (broadly this is also true because most of the standard library is unmaintained / obsolete / stable and doesn't change from nim version to version).

Where this leads, in terms of long-term planning:

• there exists a (small) subset of the standard library that is tightly integrated with the version of the language and the code generation - "broadly", this is system/ and its magics
• there exists a (small) subset of the compiler that depends on the above (small) subset of the standard library to which it is tightly coupled
• everthing else is fair game to be extracted into separately accessible components (ie std lib can be split up into constituent parts as can the compiler.
• in some distant future, it would be nice to have modern PM capabilities which treat each version of a package as a separate package, effectively allowing multiple versions of a component to be accessed from the same application - simple use case: a tool like nph wants to access multiple parser versions (1.6, 2.0 etc) to provide compatibility for source code for multiple nim versions from the same application and adapt to it

With the above long-term picture in mind, where this leads, in terms of this bug report in particular:

• it would be nice if I could depend on compiler and not have to perform any "special" tricks like $nim/... even if the experience is not as flexible as it could be (ie fully free version selection to access a particular version of the compiler source code)
• for the time being, because of the monolithic nature of both compiler and standard library, the versions could be locked in step

That limited feature set is compatible with the more modern approach from above and already brings benefits.

[jmgomez]

it would be nice if I could depend on compiler and not have to perform any "special" tricks like $nim/...

What about adding a path option to the package definition where the user can add a list of search paths that will be expanded in the nimble.path file?

[arnetheduck]

What about adding a path option to the package definition

that's what I would expect nimble to do when I say requires "compiler" (so that it propagates across transitive deps etc)

[jmgomez]

I mean, right now compiler is defined as:

 {
    "name": "compiler",
    "alias": "nim"
  },

Im referring to expand the nimble functionality so when defined like so:

 {
  "name": "compiler",
  "alias": "nim"
  "paths": ["./compiler"]
}

it will expand it in the nimble.path (paths would work in both the real and the alias)

[arnetheduck]

Im referring to expand the nimble functionality so when defined like so:

oh - didn't know about that, no idea - intuitively feels like a hack that might bridge the gap until a proper solution is found (that treats the compiler as a separate package on its own - notably, this might require more .. surgery)