In this case, you first go to Report Layout Selection to determine which report you want to create a custom layout for. If you are not a developer, just a NAV user interested in making some modifications to the layout, you can either use Microsoft Word (simpler tool, but also has lesser capabilities), or SQL Report Builder (or Visual Studio) for RDLC. You save that Report object and compile / close it, and you're done. When done, you save, close the SQL Report Builder tool, and you are returned to NAV. The SQL Report Builder and/or Visual Studio are only for the purpose of working with the report layout, toggling visibility and some basic calculations using expressions. If you need to add more, you have to go NAV (with the Report Object in Design Mode) and here you can do that.
Once in this tool, you can only work with datasets/expressions that have been declared in the NAV Report object. The default tool for designing reports (SQL Report Builder or Visual Studio) will launch, and you will be able to work with the RDL file. If you are a developer, then you will launch the NAV Development Environment, Design the report in question from the Object Designer, and then go to the menus to select View > Layout. The big difference with NAV is that when you wish to design or change a report's layout, you must first start from NAV. Keep in mind that the SQL Report Builder (or even Visual Studio) does not interact with NAV the same way that a SSRS report does. You did the right thing by skipping the report server URL.